Somebody hands you a scan result. An agency ran it, or a developer, or a tool you have never opened. It has a score, a grade, a list of findings, and a colour. What is it worth?
Four questions settle almost any automated report, and none of them require you to know anything about the tool. They are about the report's own account of itself. A report that cannot answer all four is not evidence, whatever its score says.
The worked examples below are our own engines, because those are the ones whose source we can quote. The questions are not about our tools and the answers are not a recommendation.
One: What Did It Read, and From Where?
This is the question that eliminates the most reports, and it is almost never answered on the page you are handed.
A scan run from outside, unauthenticated, reads what a stranger can reach. A scan run from inside an admin session reads what the account can see. Those are wildly different evidence bases and they are routinely presented in the same layout, with the same kind of score.
Worked example: a constant that decides everything
Our public accessibility check begins with MAX_PAGES = 1, and the comment above it is the whole disclosure: one page only, the paid audit scans six, this is the lead magnet and not the product. So the number that comes back describes a home page. It cannot describe a checkout it never loaded.
If the report you were handed does not tell you how many pages it read, that is not a detail it left out. It is the first thing you need.
The related half is the address. A scan of a domain that redirects is a scan of wherever it landed, and a report that does not say which page it graded has left you to assume. Ours states the landing address in its own note when the address it was given redirects.
Two: What Could It Not See?
Every automated check has a boundary, and the useful ones publish it in the result rather than in the marketing.
The public accessibility result carries a scope of home page, a flag for when the scan was only partial, and a note that names the six representative page types a full audit combines: home, collection, product, cart, search and policy. It says the store's overall score may differ without saying in which direction, which is the honest form of that sentence.
The clause that makes a number checkable
That same note adds that the full audit lists this same page at this same score, so a merchant can hold the two against each other. That is the difference between a number you are asked to trust and a number you can verify. When you read somebody else's report, look for the sentence that tells you how to check it. Most do not have one.
And a boundary is not always a limitation of ambition. Some of it is physics: a check that reads a public page cannot see a setting that lives behind a login, and no amount of engineering changes that. We published a whole article yesterday about what happens when a tool forgets this and reports a conclusion its evidence could not support.
Three: What Is Its Scoring Shape?
This is the question nobody asks and it changes the meaning of every number on the page. There are three shapes and they behave completely differently.
Stacking. Each finding subtracts, so ten small problems can outweigh one serious one and a long tail drags a score to zero. Flat. A category costs a fixed amount however many times it occurs. Normalised. The score is divided by how much was examined, so scanning more can change the number without anything about the store changing.
Worked example: all three, in two of our engines
Flat: our Merchant Center check subtracts a flat fifty for any prohibited-content hit, whether there is one or six, and the comment in the source says why in four words: don't stack to absurd. Missing policies subtract twenty-two for the important ones and eight for the rest, and the total is clamped so it cannot leave the nought-to-hundred range.
Normalised, and the trap in it: the accessibility score divides by the number of pages scanned. So a one-page public score is structurally harsher than the installed six-page score for the identical defect. The same two critical problems read forty-three from outside and eighty-two from inside. Neither number is wrong. A merchant comparing them without knowing this would conclude one of the two tools was lying.
So when you read a score, the question is not whether it is good. It is what would have to change for it to move. A flat penalty means fixing the second instance buys nothing. A stacking one means the long tail is where the points are. A normalised one means the number is partly a fact about the scan.
Four: Can It Express a Finding That Costs Nothing?
The subtlest of the four, and the best single test of whether a report was built to inform or to alarm.
Some things a scanner notices are simply not problems. If every finding in a report is framed as a deduction, as a risk, or as something to remediate, then the tool has no way to say we saw this and it is fine — and a tool that cannot say that will eventually tell you a correct configuration is broken, because that is the only sentence available to it.
What to look for
An informational finding that subtracts zero points and offers no fix. If the report has a category like that and uses it, somebody thought about false positives. If every line is a deduction, the scoring and the observation are the same mechanism, and you are reading a sales document with a number on it.
What a Scan Is Never Evidence Of
One thing that applies to all of them, ours included, and it is worth being blunt about.
No scan makes a store compliant with anything. Not ours, not anyone's. An automated check finds a subset of a subset: the machine-detectable instances of the rules somebody encoded, on the pages it managed to read, on the day it ran. A clean result means those specific checks did not fire. It is not a legal position, it is not an audit in the sense a regulator means, and any report that implies otherwise has told you something about its author.
If a report you were handed describes its own output as proof of compliance, that single sentence is enough to discount the rest of it, regardless of how good the underlying tool is.
The Short Version
Four questions, in order
1. What did it read, and from where? Pages, count, authenticated or not, and which address it actually landed on. 2. What could it not see, and does the report say so in its own words? 3. What is its scoring shape, and what would have to change for the number to move? 4. Can it report something that costs nothing? If not, everything it notices will arrive as a problem.
A report that answers all four might still be wrong. A report that answers none of them cannot be checked, which is a different and worse thing.
If you want the same treatment applied to specific checks rather than to the reading of them, four free scans and what each one cannot answer is written entirely around per-tool limits. AccessGuard versus accessibility overlays is the version of question four applied to a whole product category. And the rest of what we make is in seven apps and the problem each one was built for.