Case study · mango-plugins
A green result is a chain of eight claims. Break any link and the green looks exactly the same. Six of these links broke in a real eval suite — here is how each one manufactured a pass.
For a green to mean anything, every claim below must hold. Click a link to see the real defect that broke it, and the rule it left behind.
These are four real states a transcript can be in, and four real assertions from the suite. Only one state is a genuine run — but three of the four states still produce passes.
Batching buys early stopping — find a defect at job 13 instead of paying for all 126. That is real. But a batch only pays off if it survives to be summed. Every time the ruler moves, every row bought so far is void. Move the sliders.
This is a stated model, not a fitted curve: a ruler move voids every row bought since the previous move, so those rows must be re-bought in the final clean pass. The real campaign came out worse than the model — 12 rulers, 136 rows kept, ~233 discarded, about 2.7 dispatches paid per surviving row — because it also threw away a whole discovery pass. Meanwhile a single full run was measured at 16.6–20 minutes with 8 workers, and that figure sat in the project's own README the entire time.
Classified by what the fix touched. The first row is the whole question — it is the only class that justifies a behavioural eval existing at all.
| Class | Count | What the fix changed |
|---|
| Measured from git | Value |
|---|
Re-derived with git log --all --oneline -- tests/eval/
and git show <sha>:tests/eval/run.sh | wc -l. The project's CHANGELOG records
"86 vs 25" commits for the 60-day window; re-measurement gives 50 vs 28, because the retirement PR
was squash-merged and the evidence the original count came from no longer exists. A measurement
taken from commit history stops being reproducible the moment you squash.