AIV Integrity GateApache Airflow AI slop benchmark
Methodology

Methodology

Three suites, three denominators, one rule: a case only counts if AIV actually executed on it and produced a canonical receipt.

What counts as an objective defect

A finding is objective when it can be proved from the bytes of the added lines alone, with no judgement about intent or authorship. The benchmark recognises these objective classes on the added surface of a change:

ClassWhat it looks likeBlocking rule
Merge conflict blockUnresolved <<<<<<<, =======, >>>>>>> markers committed to a source fileinvariant.merge-conflict
Edit tool artifactSearch and replace or diff fence blocks that an AI editing tool uses internallyinvariant.ai-edit-artifact
PlaceholderLiteral placeholder text or unimplemented stubs on a production pathinvariant.placeholder
Attribution markerGenerated-by style comments and tool banners pasted into sourcedesign.forbidden.no-ai-generated-markers, invariant.ai-provenance
Unexpected gitlinkA new Git submodule pointer (mode 160000) not on the approved listinvariant.unexpected-gitlink
Unparseable sourceA changed file that no longer parses in its own languagesyntax.parse

Everything else that AIV reports, such as test adequacy, cohesion, density and verbosity, is advisory in the Airflow preset. Advisory findings never count as a block, and a control that only receives advisory findings is a true negative.

The three suites

Suite 1

Designed fixture agreement

50 hand built cases: 35 expected to block and 15 expected to pass. The positives reproduce the defect classes above in Airflow shaped files. The negatives include clean controls and deliberately verbose or low cohesion code that must not be blocked.

Suite 2

Production equivalent Git structure

3 real Git repositories built on disk so AIV reads them exactly as CI would: an added conflict, an unapproved gitlink, and an approved gitlink control.

Suite 3

Selected Airflow reconstruction

2 real Airflow pull requests from the corpus, rebuilt as changed file head reconstructions with patch fallback. One carries a committed merge conflict; one is a clean merged control.

How a case is scored

  1. AIV runs from the shaded CLI jar against the case repository with the Airflow preset pinned to the base commit.
  2. The JSON report must be schema version 3. Every gate result must carry a canonical result_key equal to id:partition, and partition, passed and blocks_ci must agree with each other and with the process exit code.
  3. A positive is a true positive only if the expected rule appears in the blocking partition. A block for some other reason is recorded, but does not count.
  4. A control is a true negative only if nothing blocks. Advisory findings are allowed.
  5. Any inconsistency, missing receipt or process failure is an ERROR row. Errors abort the run; they are never folded into a pass or a fail.

Labels come from added lines only

When the benchmark scans the stored Airflow patches for hard markers, it scans only lines that the patch adds. A pull request that removes an old conflict marker is doing the right thing and is not a positive. This rule removed a source of inflated true positives found in an earlier run.

Next: what AIV actually caught.

← DatasetWhat AIV caught →