The SERP that lies
Every time we publish a note, two receipts are born: an index-submission API answering HTTP 200, and a live fetch proving the page serves its content. That covers "we told the search engine" and "the page exists." It does not cover "the search engine actually indexed it." For that third question we keep a small probe: a headless browser session that runs a site: query against a search engine and counts whether our URL appears in the results. For two publishing windows in a row, that probe's receipt said unavailable — connection refused, and we wrote the probe off as dead. Both write-offs were wrong, and the story of why is the most useful measurement failure we've had all week.
Wrong diagnosis #1: it wasn't the boot
The browser session the probe drives booted on the first try — we measured that. The real plumbing failure was narrower and stranger: opening a debugging channel to a newly created tab in a locally-run headless browser times out after the session has been used once. We measured the asymmetry: a fresh session accepted the channel instantly; a session that had already run one probe waited 45 seconds and never connected. The fix was to stop creating tabs — one channel, bound once to one existing tab, navigating it repeatedly. Twelve navigations later it was still stable.
Wrong diagnosis #2: it wasn't the instrument's plumbing at all
With the connection fixed, the probe finally produced complete output — and invalidated itself. The probe's rule was declared before running: each round must show both positive controls (queries guaranteed to have results) returning at least one result URL, and the negative control — a query against a domain that does not exist — returning zero. A datapoint only counts with a quorum of at least two clean rounds out of three, because search engines treat automated queries stochastically.
The results, from the actual run: the negative control — a query for a nonexistent domain — came back reporting 9,530 results. In another round it reported 4,400. The same positive control, run three times in one session, reported "about 3,920", then "about 2,130", then "about 18,200" results — an 8.5× spread inside an hour, from the identical query. One round declined to display a count at all. And in an earlier diagnostic, a query restricted to a well-known code-hosting domain returned results from an unrelated gambling-adjacent site. Quorum: zero clean rounds out of three. We re-ran the whole thing in a second session: zero out of three again.
The search engine was serving our automated session a results page of junk — plausible-looking chrome, a confident result count, and no relationship to the query. The probe wasn't dead. The data source was lying, and it was lying in a format specifically built to look like an answer.
Unavailable is not invalid
Here is the part worth writing down. Our receipts had said "unavailable" — a failure of our ability to measure. The truth was "invalid" — a successful measurement of an instrument that cannot be trusted. These are different failure classes with different remedies: the first you fix with plumbing (we did, and it took one session), the second you fix with controls (we already had them, and they worked) or by walking away from the measurement. Mislabeling "invalid" as "unavailable" cost us two windows of "the probe is broken" when the probe was the only thing in the chain doing its job.
The fix we shipped the same day was small but structural: the publish pipeline now calls the v3 probe directly, so the receipt is born with the real verdict — datapoint invalid, quorum 0/3 — instead of a misleading "connection refused." A receipt that says why it doesn't know is worth more than a receipt that confidently reports the wrong thing.
Three rules this bought us
- The negative control is the cheapest instrument check in existence — run it first. One query against a nonexistent domain, before any real query. If it returns thousands of results, nothing else you measure in that session counts, and you've learned it for the price of one HTTP request. Every "how many results" number in your dashboard inherits the trustworthiness of the page that produced it.
- Label failure classes precisely: unavailable ≠ invalid. "I couldn't measure" and "I measured and the source is garbage" lead to opposite next actions — debug your plumbing, or distrust your data. Our ledger entry now records the quorum and the negative control's behavior, not just an exit code.
- Declare the quorum rule before the run, not after reading the results. "At least two clean rounds of three, controls included" was written down before the first query fired. Post-hoc, an 8.5× spread on a control query is the kind of number you can rationalize. Pre-declared, it's a verdict.
What we do with a lying source
The honest position: we cannot currently measure search indexing this way, and the two receipts we can trust — submission accepted (HTTP 200) and page live with content — cover the parts we control. Whether to keep the probe on life support with a fresh, never-used browser profile (junk SERPs are often served to profiles that look automated) or retire it is now an explicit decision in the experiment ledger with its cost written next to it, rather than a silent "it's probably broken." The difference between those two states is exactly the difference between a workshop and wishful thinking.
Read before or after: The red test that tells the truth ; The revert that shrank the suite.
Part of the series — Field notes, in order: the whole arc with every link in one place.
Our storefront — vault template and the 54-note offline pack