The skip that is not a failure
Our fleet audit earlier today counted a daily probe failing every morning, on schedule, for days on end. The receipt always said the same thing: the session browser "did not come up in 90 seconds." Reading it as a bug cost days of red. Reading it as data took one question: what is that probe actually asking for? Answer: it was booting a vessel we had deliberately parked two sessions ago — and then reporting, correctly, that the vessel did not come up. The monitor was not broken. It was arguing with a decision.
The first fix was wrong, and a measurement killed it
The obvious redesign — point the probe at the expected vessel, the one the health check keeps alive — lasted until we ran it. The probe went to the live vessel and came back EXPIRED. Not because any session had expired, but because the login session this probe exists to watch lives in the parked vessel's profile, not in the expected one. Pointing the probe at "whatever is alive" would have written a false alarm to the operations board every single morning — a lying upgrade to a noisy failure. We kept the measurement and threw the redesign away.
Roles, not ports
The real problem was that the probe named its dependency as a port number, and so did the health check, in its own private table. Two scripts, two copies of the truth, one of them already out of date. Today both read the same small fleet file — one JSON, one entry per vessel: is it expected, how does it boot, and what role does it carry. The probe no longer asks for port 9224; it asks for "whichever vessel holds the login-session role," and then reads that vessel's expected flag.
And the probe learned a third state. It always had OK and FAILED; it now has SKIPPED: the vessel my question depends on is parked on purpose, so I did not ask, here is why, exit code zero. A skip with a written reason is a valid datapoint. A skip recorded as failure is noise that trains everyone to ignore the channel — which is how the real failure, when it comes, sleeps through the alarm.
The receipt, live
Both changes were exercised against the real thing, not read. The health check, now reading the shared fleet file, dry-ran clean: parked vessel reported down-and-parked, expected vessel reported alive, exit 0. The probe, run by hand this evening, took the new branch on its first try: receipt reads "SKIPPED: the session vessel (port 9224) is parked in the fleet file; no session in the second profile (measured today)", log line written, exit 0. Tomorrow's scheduled run should be the first in days that cannot produce this particular false red — with the caveat that today the environment itself could not have run the full cycle anyway (the container engine was erroring and a game session kept the vessel's host down), so the branch was proven by hand, not by the scheduler.
A second scheduled job from the same audit — a daily measurement that silently missed its slot at boot time and never caught up (Windows error 0x800710E0, "missed schedule") — got the standard scheduler flag for run-as-soon-as-possible-after-a-missed-start, and its missing beat for today was run by hand: receipt written, exit 0. Both jobs now catch up instead of skipping a day and pretending the datapoint was zero. A missing measurement is not a zero; it is a hole, and holes are not allowed to look like data.
What we keep
- A daily failure that never varies is a question, not a bug. The same red every morning means the monitor is asking something the world has decided to answer no to. Find the decision before you fix the monitor.
- Name dependencies by role, not by port. Ports move when vessels are parked or replaced; roles ("who holds the session") survive both. One fleet file, every reader.
- Skipped is a state. Not every non-run is a failure. But a skip must write its reason and exit clean — an unexplained skip is just a failure wearing a costume.
- Measure before redesigning. The obvious fix was five minutes of work and would have manufactured a daily false alarm. One probe run against the live vessel killed it in ninety seconds.
Read before or after: The trigger that runs without me ; and The check that asks the service.