
Methodology
Three suites, three denominators, one rule: a case only counts if AIV actually executed on it and produced a canonical receipt.
What counts as an objective defect
A finding is objective when it can be proved from the bytes of the added lines alone, with no judgement about intent or authorship. The benchmark recognises these objective classes on the added surface of a change:
| Class | What it looks like | Blocking rule |
|---|---|---|
| Merge conflict block | Unresolved <<<<<<<, =======, >>>>>>> markers committed to a source file | invariant.merge-conflict |
| Edit tool artifact | Search and replace or diff fence blocks that an AI editing tool uses internally | invariant.ai-edit-artifact |
| Placeholder | Literal placeholder text or unimplemented stubs on a production path | invariant.placeholder |
| Attribution marker | Generated-by style comments and tool banners pasted into source | design.forbidden.no-ai-generated-markers, invariant.ai-provenance |
| Unexpected gitlink | A new Git submodule pointer (mode 160000) not on the approved list | invariant.unexpected-gitlink |
| Unparseable source | A changed file that no longer parses in its own language | syntax.parse |
Everything else that AIV reports, such as test adequacy, cohesion, density and verbosity, is advisory in the Airflow preset. Advisory findings never count as a block, and a control that only receives advisory findings is a true negative.
The three suites
Designed fixture agreement
50 hand built cases: 35 expected to block and 15 expected to pass. The positives reproduce the defect classes above in Airflow shaped files. The negatives include clean controls and deliberately verbose or low cohesion code that must not be blocked.
Production equivalent Git structure
3 real Git repositories built on disk so AIV reads them exactly as CI would: an added conflict, an unapproved gitlink, and an approved gitlink control.
Selected Airflow reconstruction
2 real Airflow pull requests from the corpus, rebuilt as changed file head reconstructions with patch fallback. One carries a committed merge conflict; one is a clean merged control.
How a case is scored
- AIV runs from the shaded CLI jar against the case repository with the Airflow preset pinned to the base commit.
- The JSON report must be schema version 3. Every gate result must carry a canonical
result_keyequal toid:partition, andpartition,passedandblocks_cimust agree with each other and with the process exit code. - A positive is a true positive only if the expected rule appears in the blocking partition. A block for some other reason is recorded, but does not count.
- A control is a true negative only if nothing blocks. Advisory findings are allowed.
- Any inconsistency, missing receipt or process failure is an
ERRORrow. Errors abort the run; they are never folded into a pass or a fail.
Labels come from added lines only
When the benchmark scans the stored Airflow patches for hard markers, it scans only lines that the patch adds. A pull request that removes an old conflict marker is doing the right thing and is not a positive. This rule removed a source of inflated true positives found in an earlier run.
Why there is no single accuracy number
The fixture suite tests agreement with a specification. The structural suite tests the Git adapter under production conditions. The selected suite tests real Airflow content. These are different questions with different sample sizes, so each gets its own confusion matrix and its own exact binomial lower bound. Pooling them would let 35 synthetic cases disguise how small the real world sample is.
Next: what AIV actually caught.