MCP Test Harness hero
PyPI version PyPI downloads Ecosystem PyPI downloads Python CI License Tests Coverage GHCR Website
v5.2.0 on PyPI + GHCR container images

The pytest-style
testing framework for MCP

Discover, run, and report on MCP server tests automatically. Functional + regression + performance testing in one tool. Replace manual MCP Inspector validation with repeatable CI-native automation.

1080+ tests · e2e dogfood
100% lib coverage
MIT licensed
No LLM required
Watch

4-section product demo

Real product pages: security docs · load guide · resiliency/chaos · HTML CI report — ~60 seconds, silent.

Productivity loop

Record → Generate → Gate

From live MCP traffic to CI gates — no manual test authoring bottleneck.

RECORD

Live calls → suite

mcp-test record captures real tool calls and writes tests + snapshots (RFC-001).

GENERATE

Scaffold from tools

mcp-test generate drafts test files from tool definitions and JSON cassettes.

GATE

CI + pre-commit

GitHub Action, mcp-test-try pre-commit hook, conformance badge, SARIF for security findings.

All packages · total downloads (all-time)

195.8k

26/26 indexed · refreshed daily from PePy / pypistats

Open integrations dashboard
Shipped & tested

What ships end-to-end

1080+ tests, 100% library coverage gate, e2e dogfood in CI. Everything below is implemented in this repo — not roadmap slides.

CLI

init, try, record, generate, experiment, conformance, export-pdf, doctor, scan-agents, audit verify

Reports

Console, JUnit, JSON, HTML dashboard, PDF export, MCP trace timeline, OSCAL assessment-results helpers

Chaos & resiliency

@marker chaos_faults, experiment catalog, throughput / load-phase SLOs, load resilience under deny

Security

SARIF · OWASP MCP · Suites A–D · incident E–K · audit-chain · AGENTS.md Unicode · load resilience · Bastion pairing

Conformance

Levels + badge (RFC-002); stateless SEP-2575 via mcp-test conformance stateless (RFC-006)

Automation

GitHub Marketplace Action, pre-commit hook, Docker GHCR, PyInstaller binary

26 PyPI packages · 195k+ downloads

Core + 25 framework shims (incl. LangGraph, Pydantic AI, OpenAI Agents) — all-time total on Integrations

Platform QA

Tool coverage map, unified portal in reports, performance baselines

Dogfood

pytest spawns real mcp-test CLI against bundled FastMCP fixtures

MCP Test Harness feature map

Visual product guide

Architecture, CI pipeline, ecosystem positioning, and assertion library — full deck with print-to-PDF.

Open features deck Ecosystem map
Dual mode: stateful MCP session vs stateless Streamable HTTP
New in 4.0.0

CI quality + manifest rug-pull gates

Deterministic merge-time controls: require security tests / severity thresholds, and snapshot the sanctioned MCP tool surface so unapproved schema or capability changes fail the PR.

manifest_gate: { enabled: true } # mcp-test.yaml mcp-test manifest check Why security-first Security testing Moat roadmap
5.2.0 · Cyber suites A–D + audit

Egress · attestation · quarantine · load resilience

Deterministic merge gates — no live model required. Suites A–D, incident E–K fixtures, hash-linked audit verify, AGENTS.md Unicode scan, and load resilience (deny-codes, no-fail-open, fairness).

  • assert_egress_quarantined · assert_capacity_enforced · assert_envelope_verified
  • assert_untrusted_context_quarantined · assert_audit_chain_intact
  • assert_control_engages_under_load · mcp-test scan-agents / audit verify
Security docs Cyber backlog
# pytest marker: security · 5.2.0
await assert_egress_quarantined(mcp, "send_message")
await assert_untrusted_context_quarantined(...)
assert_audit_chain_intact("chain.json")
# CLI
mcp-test scan-agents
mcp-test audit verify chain.json
# nightly only — not PR gates:
# -m semantic_live · -m adaptive
Visual guide

Features with diagrams

Same assets as the README — architecture, CI, transports, reports, ecosystem, and dogfood flow.

The Problem

MCP servers need real test automation

MCP Inspector is great for exploration. But you can't put a GUI in CI. You need deterministic, repeatable, code-first tests that run on every push.

3
Testing modes in one tool
Functional + Regression + Performance
12+
Built-in assertions
Protocol-aware, MCP-native
0
LLM calls needed
Deterministic, no model dependency
3
Transports supported
stdio, SSE, streamable HTTP
Three Modes, One Tool

Complete MCP test coverage

One workflow validates both answer quality and time-to-answer in CI.

FUNCTIONAL

Protocol-aware assertions

assert_tool_call, assert_resource_read, assert_prompt, assert_capabilities, assert_tool_schema, assert_protocol_version. Direct MCP protocol testing.

REGRESSION

Snapshots and determinism

assert_snapshot with ignore_fields and mask_patterns. assert_tool_idempotent for determinism checks. Catch silent regressions before they ship.

PERFORMANCE

Latency gates and load testing

assert_latency with p90/p95/p99/mean/median over N runs + warmup. assert_throughput / assert_load_phases for concurrent load, closed-loop duration, weighted mixes, and RPS. SLO-style gates in CI.

Test Report Outputs

One run. Console, JUnit, JSON, HTML — and PDF export

Like JMeter HTML dashboards, the harness ships a rich self-contained HTML report with stat cards, charts, chaos/load panels, and security findings. Export a shareable PDF summary with Save as PDF in the report or mcp-test export-pdf report.html in CI.

MCP Test Harness HTML dashboard with stat cards, charts, filters, and PDF export

Click to open the interactive HTML dashboard sample

5 Minutes to First Test

Write tests like you already know how

If you know pytest, you know MCP Test Harness. Same conventions, same discovery, same feel. The harness handles server lifecycle, MCP handshake, and session injection.

1. pip install mcp-test-harness
2. mcp-test init --server-command "python my_server.py"
3. mcp-test
# tests/test_my_server.py
from mcp_test_harness import assert_tool_call, assert_capabilities

async def test_server_has_tools(mcp_server):
    """Verify the server advertises tool capabilities."""
    await assert_capabilities(mcp_server, {"tools": {}})

async def test_echo_tool(mcp_server):
    """Call the echo tool and check it works."""
    result = await assert_tool_call(
        mcp_server, "echo", {"message": "hello"}
    )
    assert result is not None

async def test_echo_latency(mcp_server):
    """Ensure echo responds within 200ms at p95."""
    await assert_latency(
        mcp_server, "echo", {"message": "hi"},
        max_ms=200, runs=20, aggregate="p95", warmup=3
    )
Built for Production

Everything you need for MCP test automation

🔍

Auto-discovery

Finds test_*.py files and test_ functions automatically. pytest conventions. Broken files log warnings, don't crash the run.

📸

Snapshot testing

Compare responses against stored snapshots. ignore_fields for volatile data, mask_patterns for dynamic strings. Update with --update-snapshots.

Parallel execution

Multiple workers, each with its own server instance. Tests from the same file stay on one worker for fixture correctness.

👁️

Watch mode

mcp-test --watch re-runs when test files change. Configurable poll interval with debounce for rapid saves.

🏷️

Markers and filtering

@marker(timeout=60, retry=3, tags=["smoke"]). Filter with -m smoke or -k pattern. @skip(reason="...") for conditional skips.

📊

Rich reports

JUnit XML for CI. JSON with full metadata. Self-contained HTML dashboard with charts. PDF export via Save as PDF or mcp-test export-pdf.

🔌

Plugin system

Custom assertions, fixtures, reporters, transport adapters. Load via config or Python entry points (auto-discovered).

🌐

Multi-transport

stdio for local servers, SSE for remote, streamable HTTP with auth headers. Test local and remote servers identically.

🐳

Docker + GHCR

Pre-built images on ghcr.io. Runtime (:latest) and dev (:dev) targets. No local Python needed for CI.

🔐

Security testing

Suites A–D, incident E–K, audit-chain verify, AGENTS.md Unicode scan, load resilience. Pairs with MCP-Bastion for runtime defense.

Schema validation

JSON-RPC envelope checks. Post-connect validation of initialize, tools/list, resources, prompts. Best-effort call_tool probe.

🏗️

Fixture system

Built-in mcp_server (per-test) and mcp_server_session (per-module). Custom fixtures with setup/teardown. Cycle detection.

1
pip install away

No complex setup. No Java classpath. No separate test runner. One command and you're testing MCP servers.

CI
native from day one

JUnit XML, JSON reports, GitHub Action, Docker images. Designed for automated pipelines, not manual clicking.

$0
forever

MIT licensed. No per-seat, per-test, or per-server pricing. Use in any commercial product.

Responsible AI Posture

Auditable evidence for governance programs

MCP Test Harness gives teams repeatable evidence that MCP behavior is safe, reliable, and governable.

CONFORMANCE

Protocol/schema checks

Reduce unexpected behavior with automated JSON-RPC and MCP spec validation on every test run.

SECURITY

Security test packs

Injection / path / leak plus Suites A–D, audit verify, authz/schema rules. Authorization boundaries. Pairs with MCP-Bastion for runtime defense.

AUDIT

Rich run reports

Auditable artifacts for internal governance and external frameworks like the EU AI Act. Full metadata in every report.

Get Started

One pip install away

Works on Python 3.11+. Linux, macOS, Windows. Or use the Docker image.

PyPI
pip install mcp-test-harness

Core harness. Lightweight: mcp + YAML + anyio. No heavy ML dependencies.

Docker / GHCR
docker pull ghcr.io/vaquarkhan/mcp-test-harness:latest

Pre-built runtime image. No local Python needed. Tags: latest, 4.0.0, dev.