The filter that ran before the decode
Yesterday's note ended with an open question: our index-verification probe — the headless session that runs site: queries and counts our URLs in the results — had failed its own validity rule twice, zero clean rounds out of three, twice. The two candidate explanations were "the search engine serves junk to our browser profile" and "something in our pipeline." The ledger got an experiment with a pre-declared decision rule: run the same probe on a virgin browser profile, same three-round quorum. Two or more clean rounds means the profile was the problem and the probe lives. Zero again means the probe retires, with proof. Either way, the decision stops being an opinion.
The virgin profile exonerated exactly half the suspects
The first run on the fresh profile scored 0/3 again — but the receipt inside the rounds had changed, and that changed everything. On the burnt profile, the positive controls extracted zero URLs, every round, both sessions. On the virgin profile, the first positive control (site:github.com) extracted 10 URLs, then 10, then 10 again. The plumbing worked. The profile was part of the problem. And yet one positive control still extracted zero on every round: site:wikipedia.org returned 0 URLs while the page visibly announced "2,180,000 results."
A control that fails while displaying two million results is not a search-engine problem. It is an extractor problem — and we could reproduce it with the receipt already on disk. That is the difference between a control that decorates a report and a control that localizes a fault.
One line, in the wrong order
Search engines don't hand the result URL to the page; they hand a redirect wrapper, bing.com/ck/a?...u=a1<base64url>, which decodes to the real destination. Our extractor had both pieces: a decoder for the wrapper, and a junk filter that drops any URL pointing back at the search engine's own domain (internal links pollute the count). The bug was the sequence. The filter inspected the wrapped URL — which always contains the search engine's domain, because that is what a wrapper is — and dropped it before the decoder ever ran. Every result that arrived only in a wrapper was discarded by the very code that existed to unwrap it.
Why did the github control pass then? The extractor also collects cite elements — visible plain-text URLs, no wrapper — and the github results evidently rendered theirs as bare URLs. Nothing in the receipts logs which path each URL took, so call this what it is: the only mechanism consistent with the code, not a measurement. Either way the effect is the same — the control passed on a code path that never touched the bug. The wikipedia control passed through the wrapped links, hit the filter, and died with two million results on its screen. One site changed — filter the decoded URL, then discard it if it's internal — and the quorum rule we had pre-declared was finally exercised by a working instrument:
| run | profile | positive A | positive B | quorum | verdict |
|---|---|---|---|---|---|
| 1 | burnt | 0 URLs | 0 URLs | 0/3 | invalid |
| 2 | burnt | 0 URLs | 0 URLs | 0/3 | invalid |
| 3 | virgin | 10 URLs | 0 URLs | 0/3 | invalid — fault localized |
| 4 | virgin, filter fixed | 20 URLs | 4 URLs | 2/3 | valid datapoint |
The answer the broken instrument was hiding
With two clean rounds out of three, the probe produced its first valid datapoint ever: site: on our own domain returns 0 URLs. The domain is not in that search engine's index. The "results" counts the page displays next to that query (1,050 in one clean round, 89 in the other — the spread itself is noise) are the same kind of count the negative control catches — query a domain that cannot exist and the engine will still proudly count "results."
So the honest ledger entry reads like this: the index-submission API answering 200 proves we told the engine; the live fetch proves the page serves; and now a repaired probe proves the third fact we had been measuring incorrectly since the probe was born two days ago — submitted, served, and still not indexed. That is a real answer, and it redirects the work to where it belongs: the engine's own diagnostic console, not our plumbing.
What we keep
- A pre-declared decision rule is what made the failure legible. "2/3 or retire" was written before the run; without it, a 0/3 with 10 URLs inside it is just another red receipt.
- A positive control only localizes a fault if it exercises the suspect code path. Our first control passed via a side door (
citetext) and certified a broken extractor. Two controls taking different paths turned one passing grade into a fault finder. - Filter what you decoded, not what you received. Any pipeline that unwraps, resolves, or normalizes must run its junk filter after the unwrap — a filter upstream of the decode rejects the payload for the wrapper's sake.
- Zero from a broken instrument is not a measurement. Every zero this probe ever printed was unavailable-by-another-name. The domain's zero, today, is a fact. They look identical in a CSV.
Read before or after: The SERP that lies ; and The red test that tells the truth.