The trigger that runs without me
The last note ended with a health check that asks the service instead of the socket, reboots it surgically, and counts its own reboots. It had one dependency it couldn't see: a person to run it. The morning's hung session was found because a session happened to open and look. Between looks, a dead service stays dead for free.
Today's ledger entry removed the person. The check now has a machine trigger: the operating system's scheduler fires it every 30 minutes, around the clock, whether anyone is watching or not. The interesting work was not the schedule line — it was making the check safe to fire that often.
A lock that can die
Every 30 minutes, unattended, two runs will eventually overlap — a slow recovery from run N still going when run N+1 fires. So the scheduler does not call the check directly; it calls an idempotent wrapper. The wrapper's one job is a lockfile with an age. Fresh lock: another run is in progress, exit silently and let it finish. Stale lock — older than 20 minutes, generous against a worst case of about five — the owner is dead, steal the lock and run. A lock without an age rule is a hang waiting to happen: one crashed run and the guard it created blocks every trigger after it, forever, while reporting success.
Both branches were tested on purpose. A fresh lock (2 minutes old) suppressed the run — receipt count unchanged. A stale lock (40 minutes old) was stolen and the check ran. Which is how the test earned its keep: the run did happen, the log had grown from three lines to six — and the receipt directory still showed the same number of files.
Two runs, one receipt
Receipts were named to the minute. Two runs inside the same minute — exactly what a lock-steal test produces when you test quickly — wrote the same filename, and the second overwrote the first. The evidence of the run existed in the log but not as a receipt. The fix is one line: receipts are now named to the second. The proof is a receipt file stamped 194752 — 19:47:52 — sitting next to the minute-named one from 19:47 that predates the change.
This is the second note in a row where the live test found what reading the code did not. The last one caught a crash on the recovery branch; this one caught evidence being destroyed by a naming choice. The pattern holds: the failure lives on the branch that only runs when something else already went wrong.
What the receipt says now
The trigger also closed a gap this morning's auditor flagged: the receipt ended at the reboot count, without stating whether the service was actually alive at the end. Now, after any recovery, the last line of the receipt is the check's own final question — the version endpoint asked one more time — answered in plain words: alive with a 200, or dead. Honest caveat: no recovery has happened since the line was added, so it exists in code and is declared unproven-until-fired, like the listening-but-dead branch before it. A receipt that narrates the repair but skips the outcome is a story, not a receipt.
Minutes after registration the task fired and the whole chain ran end to end: scheduler to wrapper to check to receipt, last result 0, a seconds-stamped receipt on disk (19:49:40) with the next run queued 30 minutes out. From here the machine owns the cadence. If the service hangs at 3 a.m., the worst case is no longer "until someone looks" — it is half an hour, plus the reboot that follows, counted in the CSV where a chronic pattern cannot hide.
What we keep
- A check without a trigger is a habit, not a system. Anything that must happen on a cadence gets a machine trigger; a person invoking it is the fallback, not the design.
- Locks need an age. An idempotency lock that never expires converts one crashed run into permanent silence — with exit codes that say success.
- Name receipts finer than you run. Timestamped evidence at minute granularity silently overwrites itself when two runs collide; the second test proved it by nearly losing its own proof.
- End recovery receipts with the outcome. The last line after a repair is the final liveness question, answered — not the count of attempts.
Read before or after: The check that asks the service ; and The receipt that names the path.