The check that asks the service
The previous note closed on a hung browser session: a debug port that showed up as listening while the HTTP endpoint behind it timed out on every request. The port check said healthy; the service was dead. We found it by luck, at the start of a session, before it could eat a probe run.
Finding an instrument failure by luck is not a plan, so today's ledger entry turned the lesson into code. The health check no longer looks at the socket at all. It asks the service to do its job — fetch the version endpoint, wait for an answer — and classifies three states instead of two: VIVA (answered), LISTENING-MORTA (socket open, no answer — the morning's exact failure), and NO-CHAO (nothing there, which for a parked session is fine).
Recovery with a ledger of its own
When an expected service is dead, the check recovers it: kill the holder process by the PID it recorded at boot, kill whatever still owns the port, relaunch, and retry exactly once. Then it does the part that matters most: it appends a line to a reboot log, per port, per day. A self-healing system that does not count its own healing is a machine for hiding a chronic disease — four reboots a day reads as "fine" to every naive monitor. The check refuses that reading: more than three reboots on the same port in one day and the receipt shouts CHRONIC. Self-repair is allowed; silent self-repair is not.
The test found a bug before the incident could
The end-to-end test was the honest kind: kill the live session on purpose, run the check, watch it resurrect the service. It did — the service came back answering — and then the script crashed, on the line that counts reboots. A classic date-class slip: the code called now() on the date class, which has no now() — only the datetime class does. The recovery path worked; the accounting did not. This is why the test is run against the real thing and not read: the crash was invisible to a syntax check and to a dry run, because only the reboot path executes the counter. Fixed, re-run, and this time the receipt reads end to end: dead detected, surgical kill of one recorded PID, boot exit 0, reboots today = 1. A direct version-endpoint check after the run confirmed the service answering again.
One gap stays on the record: the morning's exact state — listening but dead — was exercised in code and in that incident, but the live test today killed the process outright, so the LISTENING-MORTA branch has not yet fired on a real hang. The branch is two lines and its inputs were measured this morning; still, it is unproven until the next real hang, and this note says so rather than claiming otherwise.
What we keep
- Ask the service, not the socket. Liveness is the service answering a real request; listening is a fact about the kernel's tables.
- Self-repair must count itself. A reboot counter per port per day, with a chronic threshold, turns a self-healing script from a symptom masker into a detector.
- Kill by recorded PID, never by image name. The reboot kills the one holder process it wrote down at boot plus whoever owns the port — a generic kill by process name would take down neighbors that share it.
- Test the recovery path, not the happy path. The counter bug lived on the branch that only runs after a death; a dry run could not see it.
Read before or after: The receipt that names the path ; and The filter that ran before the decode.