MCP Test Harness hero
PyPI version PyPI downloads Python CI License Tests Coverage GHCR Website
v4.0.1 on PyPI + GHCR container images

The pytest-style
testing framework for MCP

Discover, run, and report on MCP server tests automatically. Functional + regression + performance testing in one tool. Replace manual MCP Inspector validation with repeatable CI-native automation.

1080+ tests · e2e dogfood
100% lib coverage
MIT licensed
No LLM required
Productivity loop

Record → Generate → Gate

From live MCP traffic to CI gates — no manual test authoring bottleneck.

RECORD

Live calls → suite

mcp-test record captures real tool calls and writes tests + snapshots (RFC-001).

GENERATE

Scaffold from tools

mcp-test generate drafts test files from tool definitions and JSON cassettes.

GATE

CI + pre-commit

GitHub Action, mcp-test-try pre-commit hook, conformance badge, SARIF for security findings.

Shipped & tested

What ships end-to-end

1080+ tests, 100% library coverage gate, e2e dogfood in CI. Everything below is implemented in this repo — not roadmap slides.

CLI

init, try, record, generate, experiment, conformance, export-pdf, doctor

Reports

Console, JUnit, JSON, HTML dashboard, PDF export, MCP trace timeline

Chaos & resiliency

@marker chaos_faults, experiment catalog, throughput SLO params

Security

SARIF export, OWASP MCP rules, authorization boundaries, Bastion pairing

Conformance

Levels + badge (RFC-002); stateless SEP-2575 via mcp-test conformance stateless (RFC-006)

Automation

GitHub Marketplace Action, pre-commit hook, Docker GHCR, PyInstaller binary

23 PyPI packages

Core + 17 LLM/framework shims — live widget on Integrations

Platform QA

Tool coverage map, unified portal in reports, performance baselines

Dogfood

pytest spawns real mcp-test CLI against bundled FastMCP fixtures

MCP Test Harness feature map

Visual product guide

Architecture, CI pipeline, ecosystem positioning, and assertion library — full deck with print-to-PDF.

Open features deck Ecosystem map
Dual mode: stateful MCP session vs stateless Streamable HTTP
New in 4.0.0

CI quality + manifest rug-pull gates

Deterministic merge-time controls: require security tests / severity thresholds, and snapshot the sanctioned MCP tool surface so unapproved schema or capability changes fail the PR.

manifest_gate: { enabled: true } # mcp-test.yaml mcp-test manifest check Why security-first Security testing Moat roadmap
Visual guide

Features with diagrams

Same assets as the README — architecture, CI, transports, reports, ecosystem, and dogfood flow.

The Problem

MCP servers need real test automation

MCP Inspector is great for exploration. But you can't put a GUI in CI. You need deterministic, repeatable, code-first tests that run on every push.

3
Testing modes in one tool
Functional + Regression + Performance
12+
Built-in assertions
Protocol-aware, MCP-native
0
LLM calls needed
Deterministic, no model dependency
3
Transports supported
stdio, SSE, streamable HTTP
Three Modes, One Tool

Complete MCP test coverage

One workflow validates both answer quality and time-to-answer in CI.

FUNCTIONAL

Protocol-aware assertions

assert_tool_call, assert_resource_read, assert_prompt, assert_capabilities, assert_tool_schema, assert_protocol_version. Direct MCP protocol testing.

REGRESSION

Snapshots and determinism

assert_snapshot with ignore_fields and mask_patterns. assert_tool_idempotent for determinism checks. Catch silent regressions before they ship.

PERFORMANCE

Latency gates and load testing

assert_latency with p95/p99/mean/median over N runs + warmup. assert_throughput for concurrent load and RPS. SLO-style gates in CI.

Test Report Outputs

One run. Console, JUnit, JSON, HTML — and PDF export

Like JMeter HTML dashboards, the harness ships a rich self-contained HTML report with stat cards, charts, chaos/load panels, and security findings. Export a shareable PDF summary with Save as PDF in the report or mcp-test export-pdf report.html in CI.

MCP Test Harness HTML dashboard with stat cards, charts, filters, and PDF export

Click to open the interactive HTML dashboard sample

5 Minutes to First Test

Write tests like you already know how

If you know pytest, you know MCP Test Harness. Same conventions, same discovery, same feel. The harness handles server lifecycle, MCP handshake, and session injection.

1. pip install mcp-test-harness
2. mcp-test init --server-command "python my_server.py"
3. mcp-test
# tests/test_my_server.py
from mcp_test_harness import assert_tool_call, assert_capabilities

async def test_server_has_tools(mcp_server):
    """Verify the server advertises tool capabilities."""
    await assert_capabilities(mcp_server, {"tools": {}})

async def test_echo_tool(mcp_server):
    """Call the echo tool and check it works."""
    result = await assert_tool_call(
        mcp_server, "echo", {"message": "hello"}
    )
    assert result is not None

async def test_echo_latency(mcp_server):
    """Ensure echo responds within 200ms at p95."""
    await assert_latency(
        mcp_server, "echo", {"message": "hi"},
        p95_ms=200, iterations=20, warmup=3
    )
Built for Production

Everything you need for MCP test automation

🔍

Auto-discovery

Finds test_*.py files and test_ functions automatically. pytest conventions. Broken files log warnings, don't crash the run.

📸

Snapshot testing

Compare responses against stored snapshots. ignore_fields for volatile data, mask_patterns for dynamic strings. Update with --update-snapshots.

Parallel execution

Multiple workers, each with its own server instance. Tests from the same file stay on one worker for fixture correctness.

👁️

Watch mode

mcp-test --watch re-runs when test files change. Configurable poll interval with debounce for rapid saves.

🏷️

Markers and filtering

@marker(timeout=60, retry=3, tags=["smoke"]). Filter with -m smoke or -k pattern. @skip(reason="...") for conditional skips.

📊

Rich reports

JUnit XML for CI. JSON with full metadata. Self-contained HTML dashboard with charts. PDF export via Save as PDF or mcp-test export-pdf.

🔌

Plugin system

Custom assertions, fixtures, reporters, transport adapters. Load via config or Python entry points (auto-discovered).

🌐

Multi-transport

stdio for local servers, SSE for remote, streamable HTTP with auth headers. Test local and remote servers identically.

🐳

Docker + GHCR

Pre-built images on ghcr.io. Runtime (:latest) and dev (:dev) targets. No local Python needed for CI.

🔐

Security testing

assert_tool_denied, assert_authorization_boundary. MCP-aware security assertions. Pairs with MCP-Bastion for runtime defense.

Schema validation

JSON-RPC envelope checks. Post-connect validation of initialize, tools/list, resources, prompts. Best-effort call_tool probe.

🏗️

Fixture system

Built-in mcp_server (per-test) and mcp_server_session (per-module). Custom fixtures with setup/teardown. Cycle detection.

1
pip install away

No complex setup. No Java classpath. No separate test runner. One command and you're testing MCP servers.

CI
native from day one

JUnit XML, JSON reports, GitHub Action, Docker images. Designed for automated pipelines, not manual clicking.

$0
forever

MIT licensed. No per-seat, per-test, or per-server pricing. Use in any commercial product.

Responsible AI Posture

Auditable evidence for governance programs

MCP Test Harness gives teams repeatable evidence that MCP behavior is safe, reliable, and governable.

CONFORMANCE

Protocol/schema checks

Reduce unexpected behavior with automated JSON-RPC and MCP spec validation on every test run.

SECURITY

Security test packs

Catch common misuse patterns early. Authorization boundary assertions. Pairs with MCP-Bastion for runtime defense.

AUDIT

Rich run reports

Auditable artifacts for internal governance and external frameworks like the EU AI Act. Full metadata in every report.

Get Started

One pip install away

Works on Python 3.11+. Linux, macOS, Windows. Or use the Docker image.

PyPI
pip install mcp-test-harness

Core harness. Lightweight: mcp + YAML + anyio. No heavy ML dependencies.

Docker / GHCR
docker pull ghcr.io/vaquarkhan/mcp-test-harness:latest

Pre-built runtime image. No local Python needed. Tags: latest, 4.0.0, dev.