AIV Integrity GateApache Airflow AI slop benchmark
Proof

Proof

Three small designed-suite confusion matrices, their exact binomial lower bounds, and a separate full census of the stored corpus. The 35/35 numbers are not the 2,098-file run. Point estimates alone would flatter small samples, so the bounds are shown next to them.

Confusion matrices

Fixture suiteAIV resultblockedpassedexpected blockexpected pass35True positive0False negative0False positive15True negative
Fixture suite, 50 cases.
Structural suiteAIV resultblockedpassedexpected blockexpected pass2True positive0False negative0False positive1True negative
Structural suite, 3 repositories.
Selected real suiteAIV resultblockedpassedexpected blockexpected pass1True positive0False negative0False positive1True negative
Selected Airflow suite, 2 pull requests.

Recall with exact 95 percent lower bounds

Observed recall and exact 95 percent lower bound0%25%50%75%100%75% targetDesigned fixture agreement (35/35)90.0% lower boundProduction-equivalent Git structure (2/2)15.8% lower boundSelected Airflow reconstruction (1/1)2.5% lower bound
Purple dot: observed recall. Navy bar: exact Clopper Pearson 95 percent lower bound. Orange dashed line: the 75 percent target set before the run.

All three suites observed 100 percent recall and zero hard false positives. The lower bounds tell the honest story about sample size: 90.0 percent for 35 fixture positives, 15.8 percent for 2 structural positives, and 2.5 percent for a single real positive. Only the fixture suite has enough positives to clear the 75 percent target with statistical confidence. The other two are regression evidence.

Census of the stored corpus

Airflow corpus censusIndexed rows15,575Size eligible rows15,050Stored patches555Patches with added line markers1
Every indexed row and every stored patch was processed. 0 patches were missing their metadata sidecar.
MeasureValueSource file
Fixture agreement100.0%reports/labeled-latest.json
Structural regression recall100.0%results/structural-latest.json
Selected real objective recall100.0%results/selected-real-head-latest.json
Share of all real AI slop caughtunknownnot measurable from this corpus
Stored patch added line marker rate0.180%results/professional-benchmark-latest.json
Executed cases across small designed suites55sum of the three suites above — not the full corpus
Full stored-patch run (separate)2,098 files · 113 blockeddocs/data/full-stored-patch-latest.json

Executive verdict

AIV met the 75 percent point target on each small designed objective suite; the separate full stored-patch run blocked 113 of 2,098 files; broad real-world AI-slop recall remains unmeasured.

Next: why this matters for a project trying to avoid AI slop.

← What AIV caughtWhy AIV helps →