Evidence and proposals: audit chain, reports, and KIP/FLIP/SPIP checks

Author: Viquar Khan

This guide covers three things that sit around the gates: a tamper-evident record of every run, a report that turns that record into a calibration decision, and advisory checks for improvement proposals that live outside git.


1. Tamper-evident audit log

Turn it on

# .aiv/config.yaml on the protected branch
audit_log: .aiv/audit/aiv.log

or per run: --audit-log <file>, or AIV_AUDIT_LOG. Precedence: flag, then environment, then config. The config key is read from the base ref like every other key, so a pull request cannot redirect or disable its own record.

Each run appends one aiv-gate-run/1 record and prints:

[AUDIT] record #12 appended to .aiv/audit/aiv.log; chain head 1bd813b4...

The same line is added to the JSON report notices and, when the deterministic findings comment is enabled, to the PR comment.

What a record contains

Timestamp, CLI version, base and head refs and their commit SHAs, commit author, where the config came from (base-ref or workspace), exit code, every gate with its findings (rule, file, line), the notices, and an overrides object:

Field Set when
config_change The change edits .aiv/ (notice [CONFIG-CHANGE]).
skip_ignored /aiv skip was present and refused (notice [SKIP-IGNORED]).
skip_honored /aiv skip was honoured for an allowlisted, signed author (notice [SKIP-HONORED], with the email).
suppressed A baseline or inline aiv-disable removed findings (notice [SUPPRESSED], with the rule ids).

These answer the question a maintainer asks after an incident: who switched what off, and when.

Verify it

java -jar aiv-cli.jar audit verify .aiv/audit/aiv.log
java -jar aiv-cli.jar audit verify .aiv/audit/aiv.log --expect-head 1bd813b4...

Exit 0 intact, 1 tampered or anchor mismatch. A retroactive edit, deletion or reordering is reported with the index of the first broken record.

Threat model, plainly

The chain is h_i = SHA256(h_(i-1) || record_i). Without an anchor it proves internal consistency only: someone with write access to the file can rewrite it and recompute every hash. It becomes tamper-evident when the head is kept somewhere the writer does not control: the CI job log, the PR comment (GitHub keeps comment edit history), a signed tag, or a transparency log. --expect-head checks against that anchor. The log is not encrypted and holds file paths and author emails; keep it where your CI keeps artifacts. Rekor or another transparency log is not integrated.


2. Evidence report

java -jar aiv-cli.jar report --audit-log .aiv/audit/aiv.log --out aiv-report.html \
    --labels labels.csv --min-labeled 20 --export-jsonl out/

The chain is verified before anything is counted; a broken chain still produces a report with the break called out, and the command exits 1.

The report is static HTML with no script and no external requests: per-rule findings, runs, files, findings seen in blocking versus advisory gate results, reviewer labels and a calibration verdict, per-gate failures, override counts, and proposal checks.

Calibrating a rule before you make it blocking

Reviewers label findings in a CSV: rule_id,file,line,verdict with tp or fp. The verdict per rule:

Verdict Meaning
needs labels (n/m) Fewer than --min-labeled labelled findings.
ready to promote Enough labels and zero false positives: safe to move from severity: warn to fail.
keep advisory At least one false positive on an advisory rule.
review: false positives on a blocking rule A blocking rule produced a false positive; fix or demote it.
blocking, calibrated Blocking only, enough labels, zero false positives.

This is the “zero false positives on 90 days of merged pull requests” procedure from the Airflow tutorial, made mechanical.

SQL over the log (DuckDB, optional)

--export-jsonl out/ writes records.jsonl and findings.jsonl. docs/reports/aiv-audit.duckdb.sql has queries for rules by volume, findings per day per gate, overrides over time, and the latest state of each proposal. DuckDB never sits in the verdict path: the chain is the record, the JSONL is a disposable view. Apache Derby was considered and not used: it is a row store for transactions, not for this kind of aggregate.

Demo on the public bench

benchmarks/airflow-bench-compare/run_audit_demo.py runs every branch of aiv-airflow-bench with --audit-log, verifies the 89-record chain, writes the report and runs the DuckDB queries (results-audit/). The labels there are derived from the bench’s own branch names (a blocking finding on labeled/proper-* is a false positive; an invariant or design finding on labeled/slop-* is a true positive). That ground truth is circular, so the demo shows the mechanics of calibration, not the precision of any rule on real pull requests.


3. Improvement proposals (KIP, FLIP, SPIP)

Where Apache projects keep proposals

Project Proposal Lives in Decided on
Kafka KIP Confluence (cwiki.apache.org), state in a “Current state” field dev@ DISCUSS and VOTE threads
Flink FLIP Confluence, state as a page label; some pages embed Markdown dev@
Spark SPIP JIRA ticket labelled SPIP, often with a linked Google Doc dev@ vote: 72 hours, at least 3 binding +1, no binding -1
Airflow AIP Confluence dev@

The record that counts at the ASF is the mailing list. Wiki pages and documents are where the text lives.

Run it

# one page
java -jar aiv-cli.jar proposal check --template kip \
    --confluence-url "https://cwiki.apache.org/confluence/display/KAFKA/KIP-1102%3A+..." --index

# the 50 newest proposals of a space, with an audit chain and a Markdown summary
java -jar aiv-cli.jar proposal scan --template kip --space KAFKA --last 50 --index \
    --audit-log audit/proposals.log --out kip-scan/

# an SPIP in JIRA, or a file on disk
java -jar aiv-cli.jar proposal check --template spip --jira SPARK-12345
java -jar aiv-cli.jar proposal check --template-file my-template.yaml --file proposal.md

Findings are advisory: the exit code is 0 unless --fail-on-findings. Network sources use public, read-only endpoints with no credentials. Nothing is written back to the wiki, JIRA or the list.

Rules

Rule Meaning
proposal.missing-section A section the template requires is not there (numbered headings, bold or large-font paragraph headings and Markdown macros are understood).
proposal.empty-section Fewer words of its own than the template minimum, after removing template instructions.
proposal.template-text One of the template’s instruction sentences is still in place.
proposal.placeholder TBD, TODO, FIXME, XXX (uppercase), lorem ipsum.
proposal.assistant-artifact Text left by a chat assistant (“Thinking…”, “As an AI language model”, …).
proposal.missing-state, proposal.unknown-state No state, or a state the project does not use (including the template’s own “[One of …]” text).
proposal.accepted-without-thread Accepted, but no mailing-list thread is linked.
proposal.accepted-without-issue Accepted without a tracking issue, when the template requires one.
proposal.number-collision Another page in the space uses the same number (--index).
proposal.unknown-reference Cites a proposal number no page uses (--index).
proposal.changed-after-acceptance The content (status section excluded) differs from the first time the audit chain saw it accepted. The CLI refuses to compare against a broken chain.
proposal.external-doc Content is in a document AIV cannot read and that is not archived on ASF infrastructure.

Templates

Shipped: kip, flip, spip (in aiv-proposal/src/main/resources/proposal-templates/). Each records its source of truth and the calibration it went through. A project writes its own with the same keys and passes --template-file.

Sources AIV does not read

Pilot

Last 50 KIPs and last 30 FLIPs, October 2026, after one calibration pass on the same pages: 9 of 50 KIPs and 7 of 30 FLIPs have findings, and every finding was checked against the page. One KIP’s sections each contain only “Thinking…”. Details, including every false positive found and fixed, in benchmarks/proposal-pilot/README.md.