Can a deterministic gate stop AI slop before a human reviews it?
An end-to-end benchmark of the AIV Integrity Gate. 2,098 stored patch files were executed: 1,219 Apache Airflow files and 879 files from six other projects. Slides also mention 15,575 Airflow PRs as a metadata census for queue volume — not an AIV gate scoreboard.
Open the measured full stored-patch run

Read this first
All local rows and stored patches were processed. Only actual AIV executions are performance denominators. The percentage of all real AI slop caught is unknown. AIV does not detect who or what wrote the code. It blocks objective, mechanically verifiable defects such as unresolved merge conflict blocks, unauthorized Git submodule links, edit tool artifacts and placeholder text left on the added lines of a change. Disclosure, merge and closure status are context, not defect truth.
Walk the benchmark end to end
The AI slop problem
Why open source maintainers are drowning in low-effort AI generated pull requests, and what the Airflow numbers show.
Step 2Dataset
15,575 Airflow pull requests indexed over one year, plus six other projects, with merged, closed, open and disclosure counts.
Step 3Methodology
How the benchmark labels cases, what counts as an objective defect, and why each suite has its own denominator.
Step 4What AIV caught
Every blocking rule that fired, the fixture classes behind them, and the real Airflow pull request that was intercepted.
Step 5Proof
Confusion matrices, exact 95 percent lower bounds, and the full census of stored patches.
Step 6Why AIV helps
How a deterministic evidence gate keeps obvious defects out of the review queue without pretending to judge authorship.
Step 7Verification
Reproduce every number from the JSON receipts and the current source, and read the claim boundaries honestly.
Small fixture suites
These three rows are designed checks. They are not the 2,098 stored patch files. The counts above are the full stored-patch execution.
| Suite | Positives | Detected | Controls | False blocks | Exact 95% lower bound on recall |
|---|---|---|---|---|---|
| Designed fixture agreement | 35 | 35 | 15 | 0 | 90.0% |
| Production-equivalent Git structure | 2 | 2 | 1 | 0 | 15.8% |
| Selected Airflow reconstruction | 1 | 1 | 1 | 0 | 2.5% |
Each suite keeps its own denominator. Pooling them into a single field accuracy figure would be misleading, so the site never does that.