CheckoutProof has a free check on our site that reads a store's public home page and tells the merchant what it found. This is what it can honestly say about a third-party tracker from that vantage point, what it deliberately does not say, and the rule that decides the difference.
The short version: a public check can tell you which trackers a store loads. It cannot tell you whether the checkout half of any of them survived Shopify's August cutover, because every fact that would settle that lives behind the admin login. So the check asks a narrower question, and it never raises an alarm it cannot back.
What the Check Can See
A public check reads one public home page over HTTP, unauthenticated. From there it can see which scripts load. It cannot see the display scope of any script tag, it cannot see the Additional-scripts box, it cannot see the checkout profile's upgrade status, and it cannot see whether a Web Pixel exists. Those four surfaces are exactly where the cutover bit, and all four are invisible from outside.
The rule the classification encodes
Nothing observable from outside supports the sentence "this is broken", for any vendor. Not for a storefront-only integration, not for one that really does have a checkout side. A home page tells you which trackers a store loads. Every fact that would decide whether the checkout half is still firing lives behind the admin login.
So the question the classification answers is not is this broken. It is the much narrower one the evidence can actually reach: does this vendor even have a checkout counterpart worth asking about?
Two Buckets, Sixteen Trackers
Sixteen trackers, split in two. The counts below are read out of the rules file as this published.
Eleven are checkout counterparts. These normally fire a purchase event at checkout, so if that half was installed the old way it stopped without an error. They report as medium, and the wording is a question rather than a diagnosis: the finding says the storefront part is fine and still runs, that the other half is worth confirming, and that a public check cannot see any of the surfaces where the cutover actually bit.
Five are storefront-native. Their Shopify integrations put the script on the storefront by design and get checkout and purchase data another way. They report as info, they cost the store zero points, and no replacement is offered, because there is nothing to replace. The finding exists only so the merchant knows we saw the script and did not count it against them.
The severity that does not exist here
The public check has no path to HIGH at all. It emits medium and info and nothing else. HIGH belongs to the authenticated scan, which can read checkout-profile status and classify a script tag by where it actually runs — and that is the only place a claim about breakage can honestly be made.
One vendor sits in the counterpart bucket for a documented reason worth naming, because it shows the split is about install routes rather than about vendor quality: Attentive's own documentation offers a manual route that involves pasting a tag into the Additional-scripts field. A store that followed it has a checkout half. A store that used the app does not. From outside we cannot tell which, and that is exactly the case the medium finding is phrased for.
Every entry carries a why that records how the call was made: VERIFIED when a vendor's own documentation was read, INFERRED when a conservative bucket was chosen deliberately without one. Most are inferred. We would rather publish that than imply the table is finished.
Why the Check Is Built This Way
Because the first version was not. It kept one bucket, and every tracker it recognised produced a HIGH finding saying the checkout counterpart had stopped firing on the cutover date. Run against a set of real storefronts, that version raised the alarm on stores whose installs were entirely correct, most of them on one vendor's current, recommended Shopify integration, which places its script on the storefront by design and never depended on a legacy checkout script at all. There was no defect on the vendor's side. The classification above is what replaced that version, and the buckets are pinned by tests so they cannot drift back.
The specific bug was small and boring: one bucket applied to a whole table. What allowed a whole table to be treated as one bucket was that nobody had written down what the check is able to see. The list at the top of this article is that sentence, written down.
Two Traps the Fix Walked Into
Both are worth more than the bug that caused them.
Adding an enum member is a breaking change
The fix needed a new severity, info, and adding it broke three things quietly. Two modules index severity to order and rank findings. A third charged score points per tracker flag without reference to severity at all — so the new harmless finding would have been priced exactly like the old alarm, and the problem would have reproduced itself inside the paid scan while the public one looked fixed.
Grep every consumer of an enum before you add a member to it. A lookup table with no default and an accumulator that ignores the field are the two shapes to look for, and neither of them errors.
The measurement that measured us
The first sweep across the storefront set ran four requests at a time and reported that a large fraction of stores were sitting behind bot protection. They were not. We had rate-limited ourselves, and the 429s were being read as the stores refusing us.
Re-run one store at a time, six seconds apart, the picture inverted: forty-seven of forty-eight readable, and exactly one store actually blocking. A measurement tool that is also a load generator will report its own footprint as a finding about the world, and it will do it confidently.
Why a Question Beats an Alarm
The trade is worth stating plainly. HIGH is alarming, alarming converts, and a question mark does not. The check now tells a good number of merchants there is nothing here to worry about, where the alarm would have pushed them toward installing the app.
That is the correct trade and it is still a trade. A free check that cries wolf teaches merchants to ignore it, and the second time it is right nobody is listening. A free check that asks a question keeps its credibility for the day it has an answer, and the counts in this article are read from the rules file rather than remembered.
If you want the general version of this argument, four free scans and what each one cannot answer is written entirely around the limits of checks that read a public page, and it passed for the one input I happened to give it is about why a single green path is not coverage. The failures that log success is about the category both belong to. The rest of the shelf is in seven apps and the problem each one was built for.