Docs / Documentation handbook

Documentation handbook

Version: 4.0.1 Purpose: Single entry point for the full system Source markdown

Concepts, security-first rationale, CI scan report, feature map, operate recipes, and companion Bastion references.

Visual tour (start here)

CI scan report

Self-contained HTML from mcp-test --report-format html: sticky scan TOC, quality-gate scorecard, full OWASP/MCP findings table, MCP contract coverage %, portal scores. No hosted dashboard required.

Open sample report · SECURITY_TESTING.md

Scan → Test → Enforce

Diagram	What it shows
	Position vs Inspector, conformance, evals, Bastion
	Test in CI · enforce at runtime
	CLI → server → assertions → reports
	PR gate with JUnit / SARIF / HTML

Architecture at a glance

Developer / CI
      │  mcp-test
      ▼
┌────────────────────────────┐
│  MCP Test Harness          │  deterministic CI gate
│  fixtures · assertions     │  functional / regression / perf / security
│  quality_gate · manifest   │  SARIF + HTML scan report
└─────────────┬──────────────┘
              ▼
        Your MCP server
              │
              ▼  (production — companion product)
┌────────────────────────────┐
│  MCP-Bastion               │  runtime allow / block / redact / observe
└────────────────────────────┘

How to use this handbook

Repos: MCP Test Harness = CI test gate. MCP-Bastion = runtime security engine. They pair; they do not replace each other.

Part 1 — What the harness is

MCP Test Harness is a pytest-style, MCP-aware framework for Model Context Protocol servers. It is deterministic by default, CI-native, and code-first — no LLM judge in the loop.

Part 2 — Why security-first

Agentic systems turn MCP tools into programmable privilege. A single compromised tool description, missing auth boundary, or silent schema change can move data, mutate systems, or burn budget — often without a human in the loop.

Why “test later” fails for MCP

What the harness owns (CI)

• Auth boundaries (assert_tool_denied, assert_authorization_boundary)
• Payload packs (injection, path traversal, secret leak)
• OWASP-MCP / OWASP-LLM rule metadata (SARIF + HTML scan)
• Opt-in quality_gate: — require security tests, fail on severity
• Opt-in manifest_gate: — merge-time rug-pull baseline
• Contract coverage map (advertised vs tested)

What Bastion owns (runtime)

See the Bastion Documentation handbook: prompt guard, PII vault/redaction, rate/cost, RBAC, schema validation, replay, proxy boundary, local dashboard.

Scan / config (mcp-shark, Bastion scan)
        →  Test / CI gate (this harness)
        →  Enforce / runtime (MCP-Bastion)

Skipping the middle step is how teams ship green dashboards and still get surprised in production.

Part 3 — CI scan report

The HTML report is a static scan artifact, not a multi-tenant SaaS product — same nature as Bastion docs: rich evidence, no always-on platform to operate.

mcp-test --report-format html --report-output report.html
mcp-test --sarif-output findings.sarif
mcp-test --pr-summary-output mcp-pr-summary.md

# mcp-test.yaml
quality_gate:
  require_security_tests: true
  fail_on_severity: high
manifest_gate:
  enabled: true
  path: __snapshots__/mcp_manifest.snap

Part 4 — Feature map

Do not build: LLM playground, always-on hosted dashboard, out-scanning mcp-scan on general threat intel, datacenter-scale generic load farms. See ROADMAP.md.

Part 5 — Operate

Part 6 — Learning paths

1. Developer (30 min): Getting started → feature demos → one HTML report locally
2. Security reviewer: Part 2 → Security testing → enable gates → SARIF → Bastion handbook
3. Platform / perf: PERFORMANCE → baselines → RFC-006 stateless
4. Compliance: CRA_COMPLIANCE → ENTERPRISE_GOVERNANCE → evidence packs

Asset index

Related published pages & end references

This project

• Docs hub
• Documentation handbook (this page)
• Product site
• Compare · Examples · Sample scan report
• GitHub · PyPI
• Source markdown

Companion & external references