Aegis Gateway 2.0

Plug and play: Spark, Spark Connect, Kafka, Kafka Connect, and other supported sources

A project does not change its code, its dependencies or its release to be governed by Aegis. It writes (or copies) one configuration file. Aegis connects to the project the way any client would, and the agent only ever talks to Aegis.

flowchart LR
    AG["AI agent / MCP client"] -->|"MCP (one endpoint)"| AE["Aegis gateway<br/>own server or inside Knox"]
    AE -->|"type: http<br/>REST templates"| SP["Spark driver UI / History Server<br/>/api/v1"]
    AE -->|"type: spark-connect<br/>gRPC client"| SC["Spark Connect server<br/>:15002"]
    AE -->|"type: http"| KC["Kafka Connect REST<br/>:8083"]
    AE -->|"type: kafka-admin<br/>Admin client"| KB["Kafka brokers<br/>:9092"]
    AE -->|"type: mcp<br/>HTTP or stdio"| MS["Any existing MCP server"]
    AE -->|"type: http"| FL["Flink REST, Iceberg REST catalog, ..."]

Four common choices, picked by what the project already has:

The project has Source type Code in the project Shipped example
an MCP server mcp (URL or stdio command) none adopters/configurations/existing-mcp-server/existing-mcp-server.yaml
a REST API http (method, path, query, body templates) none adopters/configurations/spark/spark.yaml, adopters/configurations/flink/flink.yaml, adopters/configurations/iceberg/iceberg.yaml, adopters/configurations/kafka/kafka.yaml (Connect)
a supported Java client or non-REST protocol a ToolSourceFactory plugin jar none (the plugin lives in Aegis) jdbc, zookeeper, kafka-admin, spark-connect
an A2A 1.0 agent card a2a plugin none adopters/configurations/a2a/

Whatever the type, the operator decides every tool’s class (READ, MUTATE, DESTRUCTIVE), the typed resources it touches, who may see it, and how it is previewed. Upstream hints can only raise a class.

Apache Spark (classic): the monitoring REST API

sequenceDiagram
    participant A as Agent
    participant G as Aegis
    participant S as Spark History Server / driver UI
    A->>G: tools/call spark_list_jobs {appId}
    G->>G: exposure, schema, resource spark:app:<id>, policy, egress, rate, screening
    G->>S: GET /api/v1/applications/<id>/jobs (gateway credential)
    S-->>G: jobs JSON
    G-->>A: redacted, bounded result + decision _meta

Config: adopters/configurations/spark/spark.yaml. Point baseUrl at a History Server (:18080) or a running driver (:4040). The destructive spark_kill_submission uses the standalone master’s REST submission server, which exists only in standalone cluster mode with spark.master.rest.enabled=true; leave that source out on YARN or Kubernetes.

Validated against a real Spark 4.0 driver (RealSparkTest) and in the tutorial.

Apache Spark Connect: data and SQL, governed

sequenceDiagram
    participant A as Agent
    participant G as Aegis (Spark Connect client)
    participant S as Spark Connect server
    A->>G: tools/call spark_preview_table {table: sales.orders, limit: 5}
    G->>G: resource spark:table:sales.orders, policy, egress (http://host:15002), rate
    G->>S: gRPC ExecutePlan: table(`sales`.`orders`).limit(5).toJSON
    S-->>G: rows
    G-->>A: rows (capped by maxRows, redacted)
    A->>G: tools/call spark_sql {query} (DESTRUCTIVE)
    G-->>A: APPROVAL_REQUIRED + approval request id
    A->>G: same call, _meta dryRun=true
    G->>S: EXPLAIN <query> (analyses, never executes)
    G-->>A: plan + signed receipt
    Note over A,G: a different person approves in the admin API
    A->>G: same call + approvalToken + dryRunReceipt
    G->>S: sql(query)

Config: adopters/configurations/spark-connect/spark-connect.yaml. Needs the examples/aegis-spark-connect jar (or the ready aegis-gateway-spark-*-all.jar).

Validated against a real Spark 4.0.1 Spark Connect server in its own JVM (RealSparkConnectServerTest) and end to end in TUTORIAL-SPARK-AND-SPARK-CONNECT.md.

Apache Kafka: brokers through the Admin client

flowchart LR
    A["Agent"] -->|"kafka_delete_topic {topic}"| G["Aegis"]
    G -->|"deny: kafka:topic:__* (internal topics)"| X["POLICY_DENIED"]
    G -->|"DESTRUCTIVE: approval + receipt"| G
    G -->|"Admin.deleteTopics (gateway's SASL/SSL properties)"| K["Kafka brokers"]

Config: the kafka-admin source in adopters/configurations/kafka/kafka.yaml. kafka_create_topic uses Kafka’s own validate-only mode as its dry run (ARGUMENT validateOnly). SASL/SSL client properties go in a separate clientPropertiesFile, so no secret sits in the Aegis file. Each bootstrap server is reported to the egress control as tcp://host:port: its host joins the allow list like an HTTP baseUrl, and every call checks it and every DNS answer for it before the Admin client is used, so brokers on a private range need egress.allowCidrs. Once a permitted bootstrap broker returns cluster metadata, the Kafka client can connect to the brokers it advertises. Keep those listener addresses on the intended network and enforce the same boundary with network policy.

Kafka Connect: connectors over REST

Config: the kafka-connect source in the same kafka.yaml: connect_list_connectors, connect_connector_status (READ), connect_pause_connector (MUTATE), connect_delete_connector (DESTRUCTIVE). Validated against a real Kafka 3.9.1 Connect worker (RealKafkaTest, needs AEGIS_KAFKA_CONNECT_URL).

An existing MCP server (any project)

sources:
  - id: engine-mcp
    type: mcp
    url: https://engine.internal:9000/mcp
    era: AUTO                      # 2026-07-28 first, falls back to the 2025 handshake
    namePrefix: "engine_"
    credentials: {type: bearer, tokenEnv: ENGINE_MCP_TOKEN}
    tools:
      list_jobs: {class: READ}
      delete_job: {class: DESTRUCTIVE, resources: [{type: "engine:job", pointer: /jobId}]}
    resources: {uriPatterns: ["file:///reports/*"], allowedGroups: [analysts]}   # optional, off by default
    prompts: {names: ["summarize_*"]}                                           # optional, off by default

Unlisted tools default to DESTRUCTIVE, so nothing becomes callable by accident; unlistedTools: HIDE exposes only the listed ones.

An A2A agent (any project that publishes an agent card)

sources:
  - id: inventory-agent
    type: a2a                       # aegis-source-a2a on the classpath or in Knox ext/
    credentials: {type: bearer, tokenEnv: A2A_AGENT_TOKEN}
    settings: {cardUrl: "https://inventory-agent.internal.example/.well-known/agent-card.json"}
    tools:
      inventory_agent: {class: MUTATE}   # classify by the strongest thing the agent can do

One A2A 1.0 agent becomes one tool; Aegis is the A2A client and offers no A2A endpoint. The agent’s host must be in egress.allowHosts. Profile and limits: adopters/configurations/a2a.

Where Aegis runs

Choice When How
Own server no Knox, or a dedicated MCP edge java -jar aegis-gateway-*-all.jar --config aegis.yaml (add examples/aegis-spark-connect or use examples/aegis-gateway-spark)
Inside Apache Knox 3.0 Knox already fronts the cluster five jars (or one bundle) in ext/, AEGIS_MCP service in a topology: KNOX-EXTENSION.md
stdio a desktop client launching a local gateway --stdio with host.stdio identity (it carries no OAuth scopes, so tools with actionScopes are denied); the break-glass MCP servers include a stdio IDE setup
Container or Kubernetes a cluster deployment of the own server the repository Dockerfile and charts/aegis (StatefulSet; the image is built in CI, not published)

Source definitions and the policy/governance core are portable across all four choices, but a complete configuration is not necessarily interchangeable. Own-host and container deployments use host; Knox ignores host and requires extension/plugin jars; stdio uses host.stdio and establishes no verified OAuth scopes, so tools with actionScopes deny. Identity configuration, action-scope support, plugin packaging, and filesystem paths must match the selected host.