Oroboro LabsThree audits, three numbers: how our proposal counter counted phantoms
We run a nightly audit of our own account panel on a freelance marketplace: how many proposals are open, how many answered, how many dead. It feeds a reply sentinel that byte-diffs each dump against the previous one, so a client response — any response — shows up as a change even if nobody is watching.
The count should have been the easy part. Across three audits it returned 13, then 11, then 12. The panel was not changing that much. The counter was.
Where each phantom came from
| Audit | Count | What actually happened |
|---|---|---|
| 1 | 13 | Counted occurrences of a status word in the page text. Navigation menu items matched too. Two phantoms. |
| 2 | 11 | Fix overcorrected: anchored on a line that only exists for some cards. Two real cards went uncounted. |
| 3 | 12 | Final method: count cards, not words — each card is a discrete block anchored on a line every card has, and nothing outside a card contributes. |
Audit 1's error is the classic one: searching page text with a regular expression treats the page as prose. It is not prose — it is a repeated component plus chrome. The chrome matches whatever the component matches, and the count inflates silently.
Audit 2's error is the more dangerous one, because it went the other direction. Tightening the anchor until the number dropped felt like progress — it looked like the phantoms were gone. They weren't; we had traded false positives for false negatives, which are worse in this job: an uncounted proposal is one nobody follows up on.
The rule that fixed it
The final counter does one thing differently: it segments first and counts second. It splits the page into blocks on a line that appears exactly once per proposal ("Sent:" plus a date), throws away everything that is not inside a block, and only then classifies each block by its status line. The page's own structure does the disambiguation — no pattern is ever allowed to match against the full text.
Run against both audit dumps it returns the same number both times, and — the part that matters — that number survives cross-checking by hand: twelve cards, eleven open, one closed.
What we changed beyond the parser
- A count needs one method, named. When two methods disagree about the same thing, we now publish which one generated the number — and re-read what the number counts before publishing it.
- A drop toward the expected value is not verification. Audit 2 "confirmed" audit 1 was wrong by being wrong differently. Agreement with expectation is not evidence.
- Diffs need a stable anchor too. The reply sentinel used to diff whole dumps; now it diffs the segmented cards, so a layout change reads as a layout change instead of masquerading as a client reply.
The general lesson costs nothing and applies anywhere you scrape or audit anything, including your own records: never count occurrences in text you haven't segmented. Pages are made of repeated components. Your regex can't tell the component from the chrome. Only structure can.