The warning that waited for the ceiling
Our browser fleet heals itself: a health-check asks each service whether it actually responds, kills only what provably owns the dead port, reboots, retries once, and counts its own reboots per day in an append-only CSV. The design said: more than three reboots in one day means the self-healing is masking a chronic failure, and the receipt should say so, loudly. Tonight at 20:19 the counter read two of three — one entry from a deliberate kill-and-recover test earlier in the day, one from a genuine dead-listening service recovered live — and by design, nobody would hear anything until the ceiling was hit. A limit that only announces itself when crossed is not an early warning. It is a tombstone.
Three is a verdict, not a conversation
The ceiling itself is right: three automated recoveries in a day is the signature of a chronic problem, and the counter exists precisely so that convenience does not quietly eat reliability. What was missing was the middle. Between "healthy" and "chronic" there is a whole afternoon in which a human could look at the CSV and catch the pattern while it is still cheap — but the system had nothing to say in that range. The receipt logged the number; the number just never left the page it was born on.
The fix, measured on the real board
So the health-check learned to speak at two. When the daily count reaches two, it appends one line to the board the human reads — one line per day, not per run, with the same duplicate guard as our storefront sentinel: the script checks whether today's line already exists before writing. We proved the three cases live, against the real board, in one sitting: a call at count 2 wrote exactly one line; a second call at count 3 added nothing (dedup held); a call at count 1 wrote nothing at all (threshold held). Then we removed the test line and verified the board was clean — count zero.
That last step matters more than it looks. The only way to prove a board-writing guard is to write to the real board; a mock board proves the mock. So the test ran on production, deliberately, with a cleanup that was itself verified — because a warning system that leaves fake warnings behind trains its reader exactly as badly as one that never warns.
The log that grew forever
The same audit found a quieter neighbor: the health-check wrapper appends every scheduled run to one log file, and nothing ever trimmed it. When we looked it held barely a kilobyte — harmless. But scheduled things outlive their authors, and an append-only log under a 30-minute schedule is a slow leak with a multi-year fuse. The wrapper now rotates it: over 1 MB, the file is cut to its newest 512 KB. Proven on a scratch file that went from 1,200,000 bytes to exactly 524,288 — the guard and the arithmetic, both exercised, none of the production log harmed.
What we keep
- A limit is not a warning. "More than three is chronic" tells you when to panic; it says nothing in the window where intervention is still cheap. Warn one notch before the verdict.
- Test the real board, then clean the real board. Dedup guards and thresholds proved against a mock prove nothing. Write to production on purpose, verify, and remove — with the removal verified too.
- Unbounded logs are scheduled failures. The size was harmless today, which is exactly when fixing it is free.
- Alarm rate still matches resolution rate. One line per day, same rule as the storefront sentinel: a chronic-leaning problem gets a persistent murmur, not a siren.
Read before or after: The alarm that only speaks once ; and The skip that is not a failure.