AIV Integrity GateApache Airflow AI slop benchmark

Can a deterministic gate stop AI slop before a human reviews it?

An end-to-end benchmark of the AIV Integrity Gate. 2,098 stored patch files were executed: 1,219 Apache Airflow files and 879 files from six other projects. Slides also mention 15,575 Airflow PRs as a metadata census for queue volume — not an AIV gate scoreboard.

2,098
stored patch files executed, not a sample
113
blocked by a gate finding
1,803
passed. 182 could not be scored
1,219
Airflow patch files in that run

Open the measured full stored-patch run

Overview of the Airflow benchmark
Illustrative screenshot from the older walkthrough layout. Live KPIs above are the current full 2,098-file stored-patch run; the 35/35 table below is the separate designed fixture suite.

Read this first

All local rows and stored patches were processed. Only actual AIV executions are performance denominators. The percentage of all real AI slop caught is unknown. AIV does not detect who or what wrote the code. It blocks objective, mechanically verifiable defects such as unresolved merge conflict blocks, unauthorized Git submodule links, edit tool artifacts and placeholder text left on the added lines of a change. Disclosure, merge and closure status are context, not defect truth.

Walk the benchmark end to end

Step 1

The AI slop problem

Why open source maintainers are drowning in low-effort AI generated pull requests, and what the Airflow numbers show.

Step 2

Dataset

15,575 Airflow pull requests indexed over one year, plus six other projects, with merged, closed, open and disclosure counts.

Step 3

Methodology

How the benchmark labels cases, what counts as an objective defect, and why each suite has its own denominator.

Step 4

What AIV caught

Every blocking rule that fired, the fixture classes behind them, and the real Airflow pull request that was intercepted.

Step 5

Proof

Confusion matrices, exact 95 percent lower bounds, and the full census of stored patches.

Step 6

Why AIV helps

How a deterministic evidence gate keeps obvious defects out of the review queue without pretending to judge authorship.

Step 7

Verification

Reproduce every number from the JSON receipts and the current source, and read the claim boundaries honestly.

Small fixture suites

These three rows are designed checks. They are not the 2,098 stored patch files. The counts above are the full stored-patch execution.

SuitePositivesDetectedControlsFalse blocksExact 95% lower bound on recall
Designed fixture agreement353515090.0%
Production-equivalent Git structure221015.8%
Selected Airflow reconstruction11102.5%

Each suite keeps its own denominator. Pooling them into a single field accuracy figure would be misleading, so the site never does that.

The AI slop problem →