AIV Integrity GateApache Airflow AI slop benchmark
Dataset

Dataset

Seven open source projects, one year of pull requests, and a complete Airflow index. All counts come from the aiv-airflow-bench corpus committed alongside this repository.

Seven projects at a glance

ProjectPRs, 6 monthsPRs, 1 yearGenerated-by search, 6 monthsGenerated-by search, 1 yearStored sample, 6 monthsStored sample, 1 year
Airflow8,55215,5755,1747,474305555
Flink1,5842,4419861,0123251
Kafka1,8133,18143553238
curl1,8943,8533226571318
Godot3,6687,58947793962
Ghostty1,1922,49210273253
COSMIC compositor242425672041

Windows: one year 2025-10-22 to 2026-10-22; six months 2026-04-22 to 2026-10-22. The Generated-by search column is a GitHub search total and includes template text on some projects. It is a signal of disclosure volume, not a census of AI authored changes.

Pull requests, Generated-by hits and stored samples per project, log scaleAirflowPRs, 1 year: 15,57515,575Generated-by search, 1 year: 7,4747,474Stored sample, 1 year: 555555FlinkPRs, 1 year: 2,4412,441Generated-by search, 1 year: 1,0121,012Stored sample, 1 year: 5151KafkaPRs, 1 year: 3,1813,181Generated-by search, 1 year: 5555Stored sample, 1 year: 3838curlPRs, 1 year: 3,8533,853Generated-by search, 1 year: 657657Stored sample, 1 year: 1818GodotPRs, 1 year: 7,5897,589Generated-by search, 1 year: 7979Stored sample, 1 year: 6262GhosttyPRs, 1 year: 2,4922,492Generated-by search, 1 year: 2727Stored sample, 1 year: 5353COSMIC compositorPRs, 1 year: 425425Generated-by search, 1 year: 77Stored sample, 1 year: 4141
Per project, one year window, logarithmic scale so the smaller projects stay visible. Navy: pull requests. Orange: Generated-by search hits. Purple: stored patch sample.

Airflow: the complete index

The Airflow row is a complete index of 15,575 pull requests, not only the 555 row stored patch sample. Reading every stored body gives the following label counts. A pull request can carry more than one label.

Airflow metadata labels, one yearMerged7,265Closed unmerged2,182Open248AI disclosed4,329AI tool mention1,526AI slop discussion25
Metadata labels over 15,575 indexed Airflow pull requests. Labels overlap; they are not a partition.
MeasureLast 6 monthsOne year
Pull requests indexed8,55215,575
Filled AI disclosures3,6664,329
Filled Generated-by trailersn/a3,527
Issues in the stored index1,1002,427
Dev list messages1,9654,377
Stored patches with sidecar305555

Stored sample composition

Each project's stored sample is labelled by how the pull request was selected. These are selection classes, not quality judgements.

Stored sample by selection class, one yearAirflow (555)ai_disclosed: 311311closed_unmerged_sample: 135135merged_control: 108108objective_slop: 11Flink (51)ai_disclosed: 2424merged_control: 2121ai_tool_mention: 66Kafka (38)merged_control: 2222ai_disclosed: 99ai_tool_mention: 77curl (18)merged_control: 1515ai_slop_discussion: 33Godot (62)merged_control: 2424closed_unmerged_sample: 2424ai_slop_discussion: 1010ai_tool_mention: 44Ghostty (53)closed_unmerged_sample: 2424merged_control: 2424ai_slop_discussion: 44ai_disclosed: 11COSMIC compositor (41)closed_unmerged_sample: 2222merged_control: 1818ai_slop_discussion: 11
Stored one year sample per project (case count in parentheses), split by selection class.

Next: how the benchmark turns this corpus into verifiable tests.

← The AI slop problemMethodology →