AIV Integrity GateApache Airflow AI slop benchmark
Why AIV helps

Why AIV helps a project avoid AI slop

AIV protects review capacity by blocking objective change defects early, while keeping disclosure and subjective signals out of the hard-fail path.

What was the problem, what AIV caught, what proved it

CapabilityBeforeAfterEvidence
Objective defect interceptionLiteral and structural defects could reach expensive reviewConflict blocks and unauthorized gitlinks fail before merge35/35 fixtures, 2/2 structural, PR #59957
False-block protectionGate-wide severity could promote lower-confidence signalsPer-rule fail, warn, and off actions keep subjective signals advisory0/17 observed hard control failures
Immutable evidenceMutable workspace and movable refs could change evaluated bytesPinned commit OIDs and git show bind content to reviewed headGit adapter trust and timeout regression suites
Fail-closed executionOversized or failed collection could become an empty passMissing, oversized, malformed, and timed-out evidence stops evaluationCapture-limit and source-retrieval tests
Benchmark integrityExit failures and removed lines could inflate true positivesAdded-line labels and canonical schema v3 receipts are mandatoryReconciled TP/FN/FP/TN and zero errors
Bypass resistanceAuthor email plus generic signature status could authorize skipFull-run skip is disabled; protected rules are unsuppressibleSkip and suppression regression tests

How it fits into a review process

AIV runs where CI already runs. It reads the pull request as Git sees it, pinned to the reviewed head commit, and emits a receipt that says which rules blocked, which only warned, and which passed. Because the hard fail path contains only objective rules, a block is something the contributor fixes, not something the maintainer has to argue about. Reviewer time moves from "did you leave a conflict marker in here" to "is this the right change".

38/38
small designed suites only (35+2+1), not the 2,098-file run
0
clean changes wrongly blocked out of 17 suite controls
113
blocked on the separate full 2,098-file stored-patch run

Who benefits

BuyerPainValueProof
MaintainersReview queues absorb obvious broken changesObjective defects are intercepted before scarce reviewer time is spent38/38 on small designed suites; 113 blocks on the separate 2,098-file full run
Platform engineeringDifferent repositories enforce quality inconsistentlyOne deterministic, policy-pinned receipt works in CI, agents, and workflow gatesSchema v3, immutable OIDs, per-rule actions
Security and governanceMutable inputs and bypasses weaken evidenceFail-closed collection, unsuppressible rules, and disabled full-run skipTrust, timeout, suppression, and skip regression tests
Engineering leadershipAI policy debates lack measurable evidenceSeparates objective defects from disclosure and subjective signals0/17 observed hard control failures; broad recall reported as unknown

Recommended adoption path

  1. Start advisory: capture findings without blocking delivery.
  2. Calibrate each rule on repository-owned positives and controls.
  3. Promote only objective rules whose confidence bounds meet policy.
  4. Protect promoted rules with immutable policy and audited evidence.

What AIV is not

It is not an AI detector. It does not read disclosure checkboxes to decide anything. It does not catch a change that is syntactically perfect, well tested and wrong. Those remain human work, and this benchmark does not pretend otherwise. What AIV does is remove the mechanically obvious failures from the queue, every time, with a receipt.

Next: how to verify every number on this site yourself.

← ProofVerification →