AIV Integrity GateApache Airflow AI slop benchmark
What AIV caught

What AIV caught

Every block below is tied to a rule identifier, a file and a receipt. Nothing here is inferred from a model's opinion.

Blocking rules that fired in the fixture suite

Blocking rule occurrences across the 50 fixture casesinvariant.ai-edit-artifact21syntax.parse7design.forbidden.no-ai-generated-markers7invariant.merge-conflict6invariant.placeholder3invariant.ai-provenance1
Occurrences of each blocking rule across the 50 fixture cases. A case can trigger more than one rule, so the total exceeds 35.

Fixture classes and outcomes

Label classSubtypeCasesOutcome
mixed sampleclean plus artifact5TP
synthetic positiveedit artifact8TP
synthetic positiveattribution6TP
synthetic positivemerge conflict6TP
synthetic positiveplaceholder5TP
synthetic positivesearch replace5TP
mixed samplecohesion warn5TN
mixed sampleverbose clean5TN
synthetic negativeclean control5TN

The clean plus artifact mixed cases matter most: a mostly good change with one edit artifact buried inside still blocks. The cohesion warn and verbose clean mixed cases matter for the opposite reason: ugly but correct code passes with only advisory notes.

Production equivalent Git structure

CaseExpectedActualRulesExit
added-conflictblockblockinvariant.merge-conflict1
unexpected-gitlinkblockblockinvariant.unexpected-gitlink1
approved-gitlinkpasspassnone0

These cases are real repositories on disk. The gitlink case proves the Git adapter now reads raw diff object modes and recognises a new submodule pointer; the approved gitlink control proves the allow list works.

The real Airflow interception

Pull requestLabelOutcomeBlocking rulesAdvisory gates
#59957objective slopTPinvariant.merge-conflict, syntax.parsetest-adequacy
#70842ai disclosedTNnonenone

Pull request #59957

AIV message: Unresolved merge conflict block found on the added surface of airflow-core/src/airflow/cli/commands/connection_command.py

This is the one stored Airflow patch out of 555 whose added lines carry a hard marker. It was rebuilt as a changed file head reconstruction and AIV blocked it on both the conflict rule and a syntax parse failure, which is the expected consequence of a conflict block inside a Python file. The clean merged control, #70842, passed with advisory notes only.

What AIV did not catch, and why

The stored patch census found exactly one added line hard marker in 555 patches (0.180 percent). Objective slop is rare in a mature project with active review. Most of what maintainers call AI slop is subjective: wrong scope, no tests, hallucinated APIs, confident but wrong explanations. AIV does not claim to block those. It keeps them advisory on purpose, so that the hard fail path stays indisputable.

Next: the proof, with confusion matrices and confidence bounds.

← MethodologyProof →