KIP design vs this reference
One rule: describe the KIP as the full design; describe this repo as a control-model proof. Never claim the reference has a KIP capability listed as a gap.
| Area | This reference (proven) | KIP / Java track |
|---|---|---|
| Transport | stdio only | stdio + Streamable HTTP + OAuth |
| Backend | In-memory Kafka | Real Admin / Producer / Consumer |
| Tools / resources | 13 tools, subset of resources | ~30+ tools, ~20 resources |
| Security pipeline | 9-step fail-closed, 302/302 tests | same model in production Java |
| EOS / fencing / distributed rate / durable audit | not implemented | specified in KIP |
Slides marked KIP design are aspirational. Slides marked reference match code in this repo.
Agenda
- The Kafka MCP Server: what MCP is, why Kafka needs a first-party server
- KIP features (design) vs reference (proven)
- Security model: the fail-closed pipeline, mapped to OWASP
- Proof: 302 conformance checks on the control model
- Conclusion & questions
The Kafka MCP Server
Giving AI agents safe, governed access to Kafka.
What is MCP?
The Model Context Protocol is an open standard (announced by Anthropic, Nov 2024; donated to the Linux Foundation Agentic AI Foundation, Dec 2025). It uses JSON-RPC 2.0 in a client, host, server architecture.
Tools
Functions the model executes. State changing actions.
Resources
Read only context and data for the model.
Prompts
Templated workflows (future phase).
MCP solves the N×M integration problem: one standard interface instead of a custom connector per (data source, AI tool) pair. Already adopted across Claude, GitHub Copilot, and Google ADK.
Why agents need Kafka
- An agent deciding what to do now needs the live event history, not last night's snapshot.
- Operational streams (change events, telemetry, tool results, approvals) are the agent's memory, coordination layer, and control record.
- But today Kafka is reachable only via Java/Python client code, CLI tools, or the Connect REST API. An AI agent speaks none of these.
Motivating question: should Apache Kafka provide a first-party MCP server so agents can operate clusters through a standard protocol? The KIP answers yes.
Why it must be first-party
Wiring a Kafka cluster to an unvetted third-party MCP server means trusting a stranger with the control plane (ACLs, broker configs, reassignment) and the data plane.
The ecosystem risk
- Audit of 17 popular MCP servers: avg security score about 34/100, none declaring permissions
- Exploitable issues across roughly 7,000 reachable MCP servers
- First malicious MCP package appeared Sept 2025, undetected about two weeks
- CVEs disclosed in a widely used Git MCP server
First-party answer
- Wraps native Java clients, zero new dependencies
- Released with Kafka, stays API compatible
- Security hardening baked in, not bolted on
- Runs on vanilla Apache Kafka, Apache 2.0 licensed
Named, demonstrated attack classes: tool poisoning, rug-pulls, confused-deputy, token pass-through, indirect prompt injection.
Gaps in existing servers (KIP design)
Open-source Kafka MCP servers already exist. The KIP aims to close shared gaps; the Python reference proves the control model, not every KIP tool yet.
| Gap in existing servers | KIP-1318 (design) |
|---|---|
| Weak or missing MCP-side guardrails | 9-step fail-closed pipeline (proven in reference) |
| No ACL management | ACL tools in KIP; reference has in-memory create_acls only |
| No transactional produce (EOS) | KIP specifies produce_transactional - not in reference |
| Cloud / REST coupling | Vanilla Kafka clients (Java track) |
Capabilities (KIP design) vs reference
Reference (implemented)
- 13 tools incl. Phase-1 group offset alter/delete + create_acls recorder
- Resources: topics (+offsets), groups (+offsets/lag), cluster, audit, health
- stdio + in-memory backend + 302/302 KIP-aligned tests
KIP design (not in reference)
- Streamable HTTP + OAuth 2.1, real broker clients
- EOS / fencing, Connect suite, full group admin
- Distributed rate limits, durable audit topic
Tools in this reference (proven)
read (default allow-list)
- list_topics, describe_topic
- describe_cluster
- list / describe consumer groups
- consume_messages
mutate (opt-in)
- create_topic
- alter_topic_config
- produce_message
- alter_consumer_group_offsets
destructive (opt-in + approval)
- delete_topic
- delete_consumer_group
- create_acls (in-memory recorder)
Shipped default: read/consume only. Extra KIP tools (delete_records, Connect, EOS, …) are design-only here.
Resources (reference subset)
Served here
kafka://topics,/{name},/{name}/offsetskafka://groups,/{id},/offsets,/lagkafka://cluster,audit/recent,health
KIP design (not served here)
- offsets / lag URIs, log-dirs, metadata-quorum
- features, transactions, streams/share groups
- connectors, acls resource, …
Architecture
KIP: standalone process wrapping real Admin/Producer/Consumer. Reference: same pipeline shape over stdio + InMemoryKafka.
(host)→ Transport
stdio (HTTP: KIP)→ Security
pipeline→ Data-plane
produce / consume Control-plane
topics / ACLs
Modules use per-process circuit breakers. Connect / real brokers are Java-track work.
Transports: reference vs KIP
stdio (this reference)
- Host launches the server as a subprocess
- JSON-RPC over stdin/stdout, one local session
- Capability negotiation at initialize
- What ships and is tested today
Streamable HTTP (KIP design)
- Specified for remote / multi-replica deploy
- OAuth 2.1 / PKCE in the KIP
- Not implemented in this Python reference
- No HTTP server, no latency benchmarks here
Security architecture: layers
- Layer 1, broker auth (KIP / Java): client props to real Admin/Producer/Consumer. Reference simulates ACLs in-memory.
- Layer 2, MCP auth (HTTP, KIP): OAuth 2.1; reference validates bearer audience/issuer when configured (stdio demos).
- Authorization: broker ACLs are load-bearing; MCP guardrails complement them.
Honest by design: taint/DLP are best-effort; broker ACLs remain the load-bearing control.
The 9-step fail-closed pipeline
First denial wins. No execute-then-check paths. Around execute: input validation (-32046), rogue-agent quarantine (-32047), egress DLP on produce (-32045), sensitive-topic gating, post-execute DLP scrub, and audit. Identity comes only from the transport, never from tool arguments.
Controls in the box
Scope & access
- Topic/group prefix allow-lists (
agent.*) - readonly mode, tool allow/deny lists
- Pluggable policy engine, fail-closed
Approval & intent
- HMAC-signed, TTL-bounded tokens
- Bound to tool + resource + principal, single-use nonce
- Dry-run tools return blast radius without executing
Data protection
- DLP: 10 detectors (keys, JWT, AWS creds, PII, Luhn), redact/block/off
- Config-secret masking, bounded reads (records + bytes)
Integrity & resilience
- Taint / IFC on mutate + destructive
- Rate limits, per-module circuit breakers
- Rogue-agent kill-switch, hash-chained audit
Security guardrails: the config surface
Reference config defaults match KIP secure-by-default: read/consume allow-list. Mutate/destructive tools are opt-in. Policy is an in-process hook (no OPA HTTP client here).
| Guardrail | Property | Default (reference) |
|---|---|---|
| Tool allow / deny + readonly | tools_allowed · tools_denied · readonly | read/consume · none · false |
| Namespace scope | allowed_topic_prefixes · allowed_group_prefixes | * (tighten!) |
| Taint / IFC | taint_guard_enabled · ifc_strict | true · false |
| Approval + dry-run | approval_required_tools · dryrun_tools | delete_topic, delete_consumer_group, create_acls · none |
| Policy engine | in-process callable (KIP: optional URL) | none (fail-closed) |
| Bounded output | hard_max_records · hard_max_bytes · max_output_bytes | 100 · 1 MiB · 256 KiB |
| DLP + secret masking | dlp_mode · dlp_block_categories | redact · key/JWT/AWS |
| Rogue-agent kill-switch | max_destructive_per_minute | 10 |
| Resilience + identity | circuit_breaker_enabled · identity_propagation | true · false |
| Bearer audience/issuer | oauth_expected_audience · .issuer | none (no OAuth server) |
Mapped to OWASP Top 10 for LLM Apps (2025)
| OWASP risk | KIP-1318 control | Code |
|---|---|---|
| LLM01 Prompt Injection | Taint guard / IFC, sensitive-topic gating on consume | -32040 |
| LLM02 Sensitive Information Disclosure | DLP redact/block, config-secret masking, bounded output | -32045 |
| LLM03 Supply Chain | Prefer first-party / reviewed MCP; reference is stdlib-only core (Java track: zero new core deps) | - |
| LLM05 Improper Output Handling | Post-execute DLP scrub, MCP content + isError wrapping | - |
| LLM06 Excessive Agency | Allow/deny + readonly, prefix scope, approval gate, dry-run, kill-switch | -32041 / -32042 / -32047 |
| LLM10 Unbounded Consumption | Rate limits, circuit breakers, hard record/byte ceilings | -32029 / -32043 |
Defenses also target named MCP attack classes, shown next.
Defending MCP-specific attacks
| Attack class | What it does | Guardrail that stops it |
|---|---|---|
| Tool poisoning / rug-pull | Hidden instructions in tool metadata that change after approval | Pinned tool schemas, fixed protocolVersion, approval bound to tool + resource + principal, audit |
| Confused-deputy / token pass-through | Caller rides the server's privileges or forwards a stolen token | Identity only from transport (_identity args rejected), OAuth audience/issuer validation |
| Indirect prompt injection | Consumed record text tries to drive a destructive tool | Taint / IFC marks the session untrusted, blocks mutate/destructive (-32040), sensitive-topic gating |
| Context flooding | Oversized responses exhaust the model's context | Bounded reads: hard record + byte ceilings, cumulative output cap |
| Rogue / runaway agent | Burst of destructive calls | Per-identity kill-switch + quarantine (-32047), rate limits, circuit breakers |
First denial wins at every step; broker ACLs remain the load-bearing control underneath.
Benefits
Platform teams
- Ship Kafka access to agents with provable, fail-closed guardrails
- Community-maintained, versioned with Kafka
Agent builders
- Bounded blast radius: scope + approval + DLP + kill-switch
- Standard MCP wire shape, works with any MCP host
Security & compliance
- Fail-closed, identity from transport only
- Replay-protected approvals; in-process hash-chained audit (not a durable Kafka topic)
Operations
- Per-process rate limits + circuit breakers (reference)
- Horizontal HTTP scale is KIP design, not claimed here
Proving the guardrails hold
Every security control, run as a pass/fail test.
Validated with the MCP test suite
A declarative, multi-language test tool: one mcp-suite.yaml is compiled onto the mcp-test-harness engine and run from Python or a native runner (JUnit5, Jest, xUnit, Go). It calls the server over MCP JSON-RPC and checks results, schemas, latency, and exact error codes. Below is the live HTML dashboard from this server's run.
Proof: security controls as pass/fail tests
| Security control | Test (negative unless noted) | Expected | Result |
|---|---|---|---|
| Resource scope (prefix allow-list) | create prod-payments | deny -32041 | PASS |
| Approval gate (destructive) | delete_topic without token | deny -32042 | PASS |
| Approval gate (ACL admin) | create_acls without token | deny -32042 | PASS |
| Domain error correctness | describe / produce unknown topic | UnknownTopic | PASS |
| MCP content shape + schema (positive) | describe_cluster, list_topics, create_topic | content + schema OK | PASS |
A denial that returns the correct error code is the feature. The suite verifies the code, not just that "something threw." Broader coverage: the repo's own conformance suite (302 checks) exercises every control.
Validated with a declarative, multi-language MCP test suite.
Proven, two ways
Engine-driven
mcp-suite run → 9 / 9 passed in ~0.5s, against the native server (no shim).
Java-driven
JUnit5 adapter → runSuite 9, call("create") 3, call("approval") 2. All green.
Conclusion
- KIP-1318 is the full design (Java production track). This repo is a stdio / in-memory conformance reference for the control model.
- What is proven: a 9-step fail-closed pipeline, DLP, approval, scopes, breakers - 302/302 checks.
- Not claimed here: Streamable HTTP, live brokers, EOS/fencing, distributed rate limits, durable audit topics.
- Broker ACLs stay load-bearing; taint/DLP are best-effort complements.
Note: Python reference vs Java production
This codebase is a Python reference implementation. It is not Java and not the official Kafka MCP server.
- Purpose here: validate the security control model (stdio + in-memory + tests).
- KIP production target: Java under KAFKA-20436 (
tools/mcp-server). - Gaps (HTTP/OAuth, live brokers, EOS, Connect, distributed state, …) stay intentional in this reference.
- Those features will be added in the actual Java production code - do not describe them as shipped here.