Kafka MCP Enterprise

A fail-closed MCP conformance reference for agent workloads on Apache Kafka (KIP-1318): stdio + in-memory backend. Not Streamable HTTP, not live brokers, not the Java production module.

Start the presentation KIP-1318 walkthrough - click to open the deck
Install from PyPI KIP-1318 GitHub Test suite

Fail-closed by design

Every tools/call walks nine gates. The first denial wins. Broker ACLs stay authoritative.

01Auth
02Deny
03Allow
04Scope
05Policy
06Taint
07Approval
08Rate
09Breaker

All server features

Everything below is implemented in kafka_mcp/ and covered by the 302/302 conformance suite and/or examples, unless noted as an intentional reference gap.

What this package includes: security pipeline, MCP tools, and how it runs

A. MCP protocol & surface

JSON-RPC 2.0Strict jsonrpc: "2.0" request/response
initializeprotocolVersion, serverInfo, capabilities
tools/listVisible tools honor deny-list, allow-list, and readonly
tools/callFull fail-closed pipeline + handler; MCP content + isError
resources/list / readkafka:// catalog and topic/cluster/group/audit/health reads
stdio transportNewline-delimited JSON (kafka-mcp-enterprise)
Stateless approvalsHMAC tokens self-contained (no sticky session)
Correlation IDsPer-call corr_id on audit entries

B. Tools (13) - classified

ToolkindmoduleKafka op
list_topicsreadcontrol_planeDESCRIBE
describe_topicreadcontrol_planeDESCRIBE
describe_clusterreadcontrol_planeDESCRIBE
list_consumer_groupsreadcontrol_planeDESCRIBE
describe_consumer_groupreadcontrol_planeDESCRIBE
consume_messagesreaddata_planeREAD
create_topicmutatecontrol_planeCREATE
alter_topic_configmutatecontrol_planeALTER
produce_messagemutatedata_planeWRITE
alter_consumer_group_offsetsmutatecontrol_planeALTER
delete_topicdestructivecontrol_planeDELETE
delete_consumer_groupdestructivecontrol_planeDELETE
create_aclsdestructivecontrol_planeALTER (in-memory binding recorder in this reference)

Shipped default: six read/consume tools only (SECURE_DEFAULT_TOOLS).

C. Resources (kafka://)

kafka://topicsList topics
kafka://topics/{name}Describe topic
kafka://topics/{name}/offsetsEarliest/latest offsets per partition
kafka://clusterCluster id + brokers
kafka://groupsConsumer groups
kafka://groups/{id}Describe a consumer group
kafka://groups/{id}/offsetsCommitted offsets
kafka://groups/{id}/lagPer-topic/partition lag (KIP Phase 1)
kafka://audit/recentRecent audit entries
kafka://healthLiveness + per-module circuit breaker state

D. Fail-closed security pipeline

Every tools/call - first denial wins.

StepControlCode
1Bearer auth (audience / issuer when configured)-32001
2Deny-list (tools_denied)-32044
3Allow-list + readonly-32044
4Topic / group prefix scope-32041
5Policy engine (deny or exception → fail-closed)-32044
6Taint / IFC (destructive; approval bypasses)-32040
7HMAC approval gate-32042
8Rate limit (general vs admin)-32029
9Execute via per-module circuit breaker-32043

Also enforced: input validation, rogue quarantine, identity propagation, sensitive-topic gating, egress DLP, dry-run, consume/byte bounds, post-execute scrub, taint registration, audit.

E. Data protection (DLP)

Modesredact | block | off
Default blockprivate_key, aws_access_key, jwt
Detectors (10)email, ssn, credit_card (Luhn), phone, ipv4, aws_access_key, private_key, jwt, iban, secret_assignment
Consume / produceRedact PII on read; egress block on produce (-32045)
Config maskingPassword/secret-like keys on describe
Scrub outputsWalk entire JSON result tree

F. Approval, taint & IFC

HMAC tokensMint/verify; TTL; tool + resource + principal + nonce bind
Approval-requireddelete_topic, create_acls, and related destructive names
Taint guardBest-effort; approval can bypass; broker ACLs remain load-bearing
ifc_strictAfter untrusted read, blocks destructive/control-plane without approval

G. Scoping, exposure & identity

Allow / deny liststools_allowed defaults to read/consume only; tools_denied
Readonly modeBlocks create / produce / alter / delete / ACLs
Prefixesallowed_topic_prefixes, allowed_group_prefixes
Identity propagationOptional per-principal broker ACL check

H. Resilience & blast radius

Circuit breakersPer module: data_plane, control_plane, ecosystem
IsolationControl-plane open does not take down data-plane consume/produce
Rate limitsGeneral + admin buckets
Quarantinemax_destructive_per_minute per identity (-32047)

I. Consume semantics

No groupIdDirect Partition Assignment - no group, no rebalance
With groupIdClassic group path; rebalance counter increments

J–K. Backend & audit

In-memory KafkaTopics, produce/consume, groups, ACLs, cluster describe, dependency hooks
Audit ringHash-chained ALLOW/DENY; param truncation; kafka://audit/recent

L. Error codes (15)

-32700PARSE_ERROR -32600INVALID_REQUEST -32601METHOD_NOT_FOUND -32602INVALID_PARAMS -32603INTERNAL_ERROR -32001UNAUTHORIZED -32029RATE_LIMITED -32040TAINT_VIOLATION -32041SCOPE_VIOLATION -32042APPROVAL_REQUIRED -32043DEPENDENCY_UNAVAILABLE -32044POLICY_DENIED -32045SENSITIVE_DATA_BLOCKED -32046VALIDATION_FAILED -32047QUARANTINED

M–N. Config, quality & packaging

32 config fieldsScopes, DLP, approval secret, rate limits, IFC, hard caps, identity, OAuth audience/issuer, and more
Tests302/302 conformance + smoke + end-to-end demo
ExamplesSix scenario folders with real-world data/ fixtures
PyPIkafka-mcp-enterprise · CLI kafka-mcp-enterprise · stdlib-only core
Optional OTelpip install kafka-mcp-enterprise[otel] - not required
Agent DXAGENTS.md + Cursor / portable skills

O. Intentional reference gaps

Full HTTP listener · live brokers · HTTP policy URL client · durable audit_topic publish · wall-clock dependency_timeout_ms · production language in the KIP is Java (this package is the Python reference). See kip-alignment.md.

Install in one line

CLI entrypoint: kafka-mcp-enterprise. Import: kafka_mcp.

pip install kafka-mcp-enterprise
echo {"jsonrpc":"2.0","id":1,"method":"tools/list"} | kafka-mcp-enterprise

Proof, not marketing

Guardrails return the right error codes - not just "something failed."

Re-validation dashboard showing passing security checks

Keep exploring

Docs live in the repository. The site is packaging and orientation only.

Presentation Test report Documentation KIP-1318 KAFKA-20436