It Passed for the One Input I Happened to Give It

We shipped a tool that told merchants their store was behind bot protection when it was not. The store was wide open. We had simply refused to follow a redirect, and then blamed the merchant for our own refusal.

That is bad, and it is not the interesting part. The interesting part is that the night before, the tool had been verified end to end, and it passed. It passed honestly. It passed for the one URL it was given.

What the Bug Was

Our public store checkers take a URL from a stranger and fetch it. The fetcher made one request, and if the response was not a 200 it handed that straight back to the caller with a single explanation.

Most storefronts serve their apex as a redirect to the www host. That is not an edge case, it is the default configuration of a large share of the web. Verified again the morning this published: brooklinen.com and allbirds.com both answer their bare domain with a 301, under our own client and under a browser alike, so bot protection was never involved.

So the tool failed on the address a merchant actually types. Not an unusual address. The obvious one.

The Verification That Passed

Here is the sentence worth keeping. A single green path is not coverage.

The tool was exercised end to end. A URL went in, a real scan came back, findings rendered. Everything a person would think to check was checked, and every one of those checks was correct. The verification was not sloppy. It was narrow in a way that was invisible from inside it, because the one input happened to be a store that did not redirect.

The fix for that is not more care. It is varying the shape of the input before declaring anything works: apex versus www versus the myshopify host. Three shapes, one minute, and the bug is impossible to miss.

We Had Already Published the Right Answer

This is the part that stings.

Seventeen days before the bug was found, we published a guide to fetching a URL a stranger typed. It contains this, verbatim:

If your use case genuinely requires following redirects, the rule is
mechanical: every hop goes back through the full resolve-validate-pin
cycle as if freshly typed. There is no shortcut where hop two inherits
hop one's validation.

That is exactly the fix that shipped. The article named the correct mechanism, in the correct detail, and then said we did not need it — because “for a checker, a redirect is a finding, not an instruction.”

Seventeen days later that sentence was failing on the majority of stores. Redirects are now followed manually, at most twice, with the address re-validated and the connection re-pinned on every hop. The library's own redirect handling stays off, because turning it on would reuse the first host's pinned address and skip the check entirely, quietly converting the guard into a hole.

The Test That Survived the Change It Existed to Catch

The same guide recommended a check for keeping the pattern honest:

assert follow_redirects is off at every call site
(a one-line grep in CI keeps all four copies honest)

Every call site still sets it off today. That grep passes cleanly, and it says nothing at all, because three of those sites now follow redirects in a loop of their own. The flag stayed the same while the behaviour inverted.

An assertion can outlive the condition that justified it. This one would have gone on passing for years, reported as evidence, while describing a world that no longer existed.

What We Are Not Claiming

Not that we got there quickly. The same rule sat unapplied one engine over for two days after we knew it, in code we own.

The rule the fix encodes is that a non-200 is not one thing: a redirect, a bot challenge, and a dead host are three different facts about a merchant's store and deserve three different sentences. Three engines were fixed on the day. The fourth was not, because its scan was never broken — a different code path follows the redirect — so all that was wrong was the explanation attached to a correct result. It went on telling merchants on the majority store configuration that their store challenged our first request when it had done nothing of the kind, and stamping a complete audit as partial, for two more days. It now says which page it actually audited instead. A correct result with a false explanation is still a false thing to say to somebody, and being able to see that and still leave it sitting there is the more honest lesson of the week.

Three Things Worth Stealing

1. Vary the shape of the input, not just the value. One valid URL proves one valid URL. Apex, www and the platform host are three shapes and they take a minute.
2. Never write a single explanation for a category of failure. if status != 200 followed by one sentence will eventually tell somebody something false about their own system, confidently.
3. Assert the behaviour, not the flag. A grep for a setting survives a rewrite that inverts what the setting means. Give the code a host that redirects and prove the second hop was independently validated.

None of these is clever. All three were available to us before any of this, and one of them we had already written down and published. The gap between knowing a rule and having applied it is where this kind of bug lives, and the only reliable way across it is to check the thing rather than the description of the thing. Which is the same conclusion as the code that logs success while failing, arrived at from the opposite direction: there, the report was wrong about the work; here, the check was right about the wrong question.