A single green path is not coverage. A tool can be verified end to end, pass honestly, and have proved only that it works for the one input it happened to be given. This is the anatomy of that gap, with our own public store checkers as the worked example, and the three habits that close it.
The checkers take a URL a stranger typed and fetch it. Most storefronts answer their bare domain with a redirect to the www host; that is not an edge case, it is the default configuration of a large share of the web. Verified again the morning this published: brooklinen.com and allbirds.com both answer their bare domain with a 301, under our own client and under a browser alike. So the address a merchant actually types is a redirect, and a fetcher that treats every non-200 as one kind of failure will misread the most ordinary address on the web.
What the Checkers Do Now
Redirects are followed manually, at most twice, with the address re-validated and the connection re-pinned on every hop. The library's own redirect handling stays off, because turning it on would reuse the first host's pinned address and skip the check entirely, quietly converting the guard into a hole.
And a non-200 is no longer one thing. A redirect, a bot challenge and a dead host are three different facts about a merchant's store, and each gets its own sentence. The checker that finds a challenge says so; the checker that finds a redirect follows it; the checker that finds nothing at all says which address it tried.
Why the fetcher is built this way
The first version made one request and handed any non-200 back to the caller with a single explanation. Two callers shared it. On a redirect, the free checker reported a security challenge that was not there, and the paid scan recorded the storefront as unchecked and contributed no findings, which on a results page is indistinguishable from a clean storefront. Both had been verified the night before, for one URL that happened not to redirect.
The Verification That Passed
Here is the sentence worth keeping. A single green path is not coverage.
The tool was exercised end to end. A URL went in, a real scan came back, findings rendered. Everything a person would think to check was checked, and every one of those checks was correct. The verification was not sloppy. It was narrow in a way that was invisible from inside it, because the one input happened to be a store that did not redirect.
The fix for that is not more care. It is varying the shape of the input before declaring anything works: apex versus www versus the myshopify host. Three shapes, one minute, and the gap is impossible to miss.
The Guide Already Had the Mechanism
Seventeen days before the change, we published a guide to fetching a URL a stranger typed. It contains this, verbatim:
If your use case genuinely requires following redirects, the rule is
mechanical: every hop goes back through the full resolve-validate-pin
cycle as if freshly typed. There is no shortcut where hop two inherits
hop one's validation.That is exactly the mechanism that shipped. The guide named it, in the correct detail, and then judged that a checker did not need it, because “for a checker, a redirect is a finding, not an instruction.” The majority store configuration settled the question the other way, and the mechanism the guide had already described is the one the checkers now use.
The Test That Survived the Change It Existed to Catch
The same guide recommended a check for keeping the pattern honest:
assert follow_redirects is off at every call site
(a one-line grep in CI keeps all four copies honest)Every call site still sets it off today. That grep passes cleanly, and it says nothing at all, because three of those sites now follow redirects in a loop of their own. The flag stayed the same while the behaviour inverted.
An assertion can outlive the condition that justified it. This one would have gone on passing for years, reported as evidence, while describing a world that no longer existed.
A green result is only evidence if you know what would make it red
The regression tests written for the change nearly repeated the same error in miniature. Their fixture resolved every hostname to 203.0.113.10 — which looks like a test address and is documentation space, not test space. Python's own is_global returns false for it, so the SSRF guard rejected it and every resolve failed. Three tests asserting that a bad redirect is refused then passed without exercising anything at all. They were green because nothing ran. Fixed by using a genuinely public address, and by making every refusal test prove the first hop actually happened before checking that the second was refused.
The Fourth Engine
Three engines changed on the day. The fourth's scan was never affected, because a different code path follows the redirect, so all that needed changing was the explanation attached to a correct result: it had described a challenge that had not happened and marked a complete audit as partial. It now names the page it actually audited. A correct result deserves a correct explanation, and the two days that fix waited are the honest measure of how easy it is to leave a wrong sentence next to a right number.
Three Things Worth Stealing
- Vary the shape of the input, not just the value. One valid URL proves one valid URL. Apex,
wwwand the platform host are three shapes and they take a minute. - Never write a single explanation for a category of failure.
if status != 200followed by one sentence will eventually tell somebody something false about their own system, confidently. - Assert the behaviour, not the flag. A grep for a setting survives a rewrite that inverts what the setting means. Give the code a host that redirects and prove the second hop was independently validated.
None of these is clever. All three were available before any of this, and one of them was already written down and published. The gap between knowing a rule and having applied it is where this kind of gap lives, and the only reliable way across it is to check the thing rather than the description of the thing. Which is the same conclusion as the code that logs success while failing, arrived at from the opposite direction: there, the report was wrong about the work; here, the check was right about the wrong question.
The Free Checkers, and What Each One Cannot Answer
Four public scans you can run on your own store, written around their limits rather than their reach — including the one this article is about.
Four Free Scans