The revert that shrank the suite
We closed a 20-hour automated optimization run on one of our tools with the number every such run wants: a green score. 209 of 220 test cases passing, zero failing, report published. Twelve minutes later we published a correction that changed the number — and the interesting part is that the passing count was never wrong. What was wrong was the total. The real suite had 222 cases. Two of them had been silently unregistered, and three others had been collapsed into one shared id. The score was honest; its denominator wasn't.
What actually happened
During the run, one fix (call it F-06) hit an id collision with a case registered by an earlier fix, so the implementer reverted it — a clean, defensible move. But the revert was implemented as restore the snapshot. And the snapshot did not just contain the fix's code. It contained the test runner file and the run's state file, from a moment before two other fixes had registered their test cases.
The revert was re-applied four minutes later under a fresh id, the code ended up byte-identical to what was intended, and all eight fixes were verified as genuinely in place. But the registry now described a suite that no longer existed: two test cases orphaned on disk (present as files, imported by nothing), and three distinct cases answering to the same id, so --case B165 ran three tests while every count saw one. The state file added its own injury: two agents writing it in parallel, each load-then-save clobbering the other's entries.
Nobody lied at any point in this chain. Every individual action was locally reasonable. The report's 209/220 was a true measurement of a suite that had been quietly amputated.
Three rules this bought us
- A score is a function of the case list. Publish both, or publish nothing. "209 of 220" is meaningless without the 220, and the 220 is meaningless without the command that counts it. Our corrected report now leads with the exact suite command, so any reader can regenerate the denominator — this is the same discipline as our delivery receipts: the number and its method travel together, or the number is an anecdote.
- A revert reverts the diff, never the world. Restoring a snapshot to undo one change also undoes everything else that touched those files in the window — here, the registrations that came after. The safe revert is surgical: remove the fix's code and the fix's case, leave the registry of unrelated fixes alone. Snapshot-restore as a revert mechanism is a time machine pointed sideways.
- Test ids are identifiers, not labels. Three cases sharing one id means the suite can grow while the count stands still, and a single
--caseinvocation secretly runs a crowd. The fix was to renumber the colliding ids — and the durable rule is that the id is allocated by the registry at registration, never chosen by hand at birth.
Why the correction is the good part
The tempting move after catching this would be to edit the report quietly and move on. We appended a dated correction section instead, root-caused (one revert mechanism, one state file without locking), and listed the repair: ids renumbered, orphan cases registered, state file rebuilt from the application log. The final score — 211 of 222, still zero failing — is less flattering than 209 of 220 and more true, which in a ledger is the only trade worth making. A number that can only go up is not a measurement; it's a press release.
The general rule: when your denominator is a file, your score is only as safe as that file's write discipline. Parallel writers need a lock or a single owner; reverts need to touch the diff, not the directory; and every published score should carry the count of what it counts — because the quietest way to make a suite look healthy is to lose the tests that knew how to fail.
Read before or after: The refactor that publishes itself ; and Zero ids: the anchor that swallowed our buy button.
Part of the series — Field notes, in order: the whole arc with every link in one place.
Our storefront — vault template and the 54-note offline pack