CheckoutProof has a free check on our site that reads a store's public home page and tells the merchant what it found. For a while, what it told them was wrong, and it was wrong in the most expensive direction available: it said something was broken when nothing was.
This is what it said, why it was false, and the rule the fix encodes. It is also, unavoidably, an article about a tool of ours being wrong, which is the only honest way to write about the fix.
What It Said
The check looks for known third-party trackers loading on the storefront. Every match produced a HIGH finding, and the HIGH said that the tracker's checkout and order-status counterparts stopped firing on 26 August 2026, the date Shopify shut off the legacy checkout surfaces.
We ran it against a set of verified merchant storefronts to see what it was actually producing. Of the twenty-eight stores that answered, sixteen were told something was broken. And fourteen of those sixteen were flagged on a single vendor.
Why it was false
The vendor was Klaviyo, and the script the check was finding is static.klaviyo.com/onsite/js/<id>/klaviyo.js. That is Klaviyo's current, correct Shopify install. It is placed by their Shopify integration or their app embed, its behavioural events arrive through a Shopify pixel, and its order data comes through the integration's own sync. Nothing about it depends on a legacy checkout script, so the August cutover could not have touched it.
There is no defect on Klaviyo's side here, and there never was. The install we were flagging is the one they tell merchants to use. The error was entirely ours.
The Mistake Underneath It
The specific bug is easy to state and boring: one bucket, applied to every vendor in the table. The interesting part is what allowed a whole table to be treated as one bucket, which is that we had not written down what the check is able to see.
A public check reads one public home page over HTTP, unauthenticated. From there it cannot see the display scope of any script tag, it cannot see the Additional-scripts box, it cannot see the checkout profile's upgrade status, and it cannot see whether a Web Pixel exists.
The rule the fix encodes
Nothing observable from outside supports the sentence "this is broken", for any vendor. Not for Klaviyo, not for the ones that really do have a checkout side. A home page tells you which trackers a store loads. Every fact that would decide whether the checkout half survived the cutover lives behind the admin login.
So the question the classification answers is no longer is this broken. It is the much narrower one the evidence can actually reach: does this vendor even have a checkout counterpart worth asking about?
What Replaced It
Sixteen trackers, split in two. The counts below are read out of the rules file as this published.
Eleven are checkout counterparts. These normally fire a purchase event at checkout, so if that half was installed the old way it stopped without an error. They report as medium, and the wording is a question rather than a diagnosis: the finding says the storefront part is fine and still runs, that the other half is worth confirming, and that a public check cannot see any of the surfaces where the cutover actually bit.
Five are storefront-native. Their Shopify integrations put the script on the storefront by design and get checkout and purchase data another way. They report as info, they cost the store zero points, and no replacement is offered, because there is nothing to replace. The finding exists only so the merchant knows we saw the script and did not count it against them.
The severity that no longer exists here
The public check now has no path to HIGH at all. It emits medium and info and nothing else. HIGH belongs to the authenticated scan, which can read checkout-profile status and classify a script tag by where it actually runs — and that is the only place a claim about breakage can honestly be made.
One vendor sits in the counterpart bucket for a documented reason worth naming, because it shows the split is about install routes rather than about vendor quality: Attentive's own documentation offers a manual route that involves pasting a tag into the Additional-scripts field. A store that followed it has a checkout half. A store that used the app does not. From outside we cannot tell which, and that is exactly the case the medium finding is phrased for.
Every entry now carries a why that records how the call was made: VERIFIED when a vendor's own documentation was read, INFERRED when a conservative bucket was chosen deliberately without one. Most are inferred. We would rather publish that than imply the table is finished.
Two Traps the Fix Walked Into
Both are worth more than the bug that caused them.
Adding an enum member is a breaking change
The fix needed a new severity, info, and adding it broke three things quietly. Two modules index severity to order and rank findings. A third charged score points per tracker flag without reference to severity at all — so the new harmless finding would have been priced exactly like the old false alarm, and the defect would have reproduced itself inside the paid scan while the public one looked fixed.
Grep every consumer of an enum before you add a member to it. A lookup table with no default and an accumulator that ignores the field are the two shapes to look for, and neither of them errors.
The measurement that measured us
The first sweep across the merchant set ran four requests at a time and reported that a large fraction of stores were sitting behind bot protection. They were not. We had rate-limited ourselves, and the 429s were being read as the stores refusing us.
Re-run one store at a time, six seconds apart, the picture inverted: forty-seven of forty-eight readable, and exactly one store actually blocking. A measurement tool that is also a load generator will report its own footprint as a finding about the world, and it will do it confidently.
What This Costs Us to Say
The uncomfortable part is not the bug. It is that the tool was more useful when it was wrong: HIGH is alarming, alarming converts, and a question mark does not. We replaced a finding that pushed merchants toward installing the app with one that tells a good number of them there is nothing here to worry about.
That is the correct trade and it is still a trade. A free check that cries wolf teaches merchants to ignore it, and the second time it is right nobody is listening. The classification is now pinned by tests so the buckets cannot drift back, and the counts in this article are read from the rules file rather than remembered.
If you want the general version of this argument, four free scans and what each one cannot answer is written entirely around the limits of checks that read a public page, and it passed for the one input I happened to give it is the same failure in a different tool. The failures that log success is about the category both belong to. The rest of the shelf is in seven apps and the problem each one was built for.