How to Fetch a URL a Stranger Typed: SSRF, DNS Rebinding, and the Transport That Pins

Our free checkers have one job: a merchant pastes their storefront URL, our server fetches that page, and an engine grades what it finds. Which means we operate, on purpose, the textbook setup for server-side request forgery: an unauthenticated endpoint that makes our server issue an HTTP request to wherever a stranger points it. Re-counted this morning: three checker pages on this site feed four public endpoints — the accessibility, Made-in-USA, and Merchant Center checkers, plus CheckoutProof's readiness endpoint, which is live and open ahead of its page. None requires a login, because they're lead magnets; all four run the same guard, because a stranger's URL is the most hostile input a backend accepts. The scopes essay spent one clause on that guard. This is the whole mechanism — including the step most SSRF write-ups skip, the census of our own copies (less flattering than it sounds), and the one fetch path where the pattern can't apply.

Why the Obvious Fix Isn't One

Everyone's first SSRF defense looks the same: parse the URL, resolve the hostname, check the addresses against the private ranges, and if they're clean — fetch the URL. It reads airtight and it has a hole through the middle: after your check passes, the HTTP client resolves the hostname again. Your validation ran against one DNS answer; the connection uses another. An attacker who controls a domain's DNS can answer your validator with a harmless public address and your client with an internal one — the classic DNS rebinding move, a time-of-check-to-time-of-use gap wearing a network costume. The uncomfortable summary: validating a hostname and then fetching the hostname is not a validation at all. Whatever you validated is not the thing you used.

So the whole pattern falls out of one sentence: resolve once, validate what you resolved, and connect to exactly that. Three steps, each with a trap of its own.

Step 1: Resolve Once, Validate Every Answer

The validator in our engines does its whole job before any request exists. Reject anything that isn't http or https. Reject any port besides 80 and 443 — a URL is allowed to name a port, and an internal service is allowed to listen on one. Then resolve the hostname once with getaddrinfo, and check every address it returns, because a hostname is a list, not a value — one clean address in the answer set proves nothing about its siblings. The predicate, from the source:

ip = ipaddress.ip_address(addr)
if (not ip.is_global
        or ip.is_private or ip.is_loopback
        or ip.is_link_local or ip.is_reserved
        or ip.is_multicast or ip.is_unspecified):
    raise ValueError("resolves to a non-public address")  # any bad answer kills the whole request

Two properties worth noticing. First, this is a deny predicate over resolved addresses, not an allowlist — there is no roster of permitted hosts anywhere, because a public checker's whole promise is "any storefront." (We'd earlier described this as an "allowlist" in one clause of the scopes essay; that word was wrong and we've corrected it there.) Second, the validator finishes by forcing the origin to https regardless of what was typed, and returning two things it has bound together: the hostname, and the single already-validated IP the connection will use — preferring the IPv4 answer for the pin.

Step 2: Pin the Connection, Keep the Name

Now the part that makes the pattern actually work, and the reason this article exists. You cannot just fetch https://203.0.113.7/ in place of the hostname — TLS would try to verify a certificate against an IP, and virtual hosting would serve you the wrong site. You need the connection to go to the validated IP while everything cryptographic and semantic still speaks the name the user typed. In httpx, that's a custom transport — ours subclasses AsyncHTTPTransport and rewrites each outbound request:

class PinnedTransport(httpx.AsyncHTTPTransport):
    # Connect to a pre-validated IP so httpx can't independently re-resolve
    # the hostname (closes the DNS-rebinding / TOCTOU gap). SNI, certificate
    # verification, and the Host header still use the real hostname.
    async def handle_async_request(self, request):
        request.extensions["sni_hostname"] = self._host   # TLS handshake: real name
        request.headers["Host"] = self._host              # virtual hosting: real name
        request.url = request.url.copy_with(host=self._ip) # the socket: pinned IP
        return await super().handle_async_request(request)

Three lines, three layers deliberately split apart: the socket dials the address you validated; the TLS handshake presents the typed hostname via SNI so the server offers the right certificate and verification checks the right name; and the Host header keeps virtual hosting honest. The client is never given the chance to resolve anything, because the URL it sees already contains an IP. That's the whole trick — the check and the use are now the same DNS answer.

Step 3: Refuse Redirects — the Half Everyone Omits

A pinned first request is only half a fix, because the response can say “go here instead.” Follow it, and your carefully pinned client issues a brand-new request to a brand-new hostname that was never validated at all — the redirect is a second stranger-typed URL, delivered by the first one. All four of our call sites set follow_redirects=False, verified again this morning. A storefront that redirects gets graded on what it serves at the address the user gave us, and that's the honest deal: for a checker, a redirect is a finding, not an instruction. If your use case genuinely requires following redirects, the rule is mechanical — every hop goes back through the full resolve-validate-pin cycle as if freshly typed. There is no shortcut where hop two inherits hop one's validation.

The Census: One Design, Four Copies

Time for the disclosure this site owes you whenever it describes its own code. We re-ran the comparison this morning by AST extraction and normalized hashing across all five modules that fetch stranger-typed URLs, and the result is not “four independent implementations agree” — it's better and worse than that: four of the five are the same code. The four transports hash identically; the four validators hash identically; two of them are byte-for-byte identical files' worth of logic, and a third differs only in a leading underscore because it's meant to be imported. One design, written once, copy-pasted into four engines. The honest consequence is about maintenance, not security: a fix to the predicate has to land in four files, and there is no shared module to land it in once. What does differ per copy is the operational envelope: each endpoint carries a response-size cap and a tight timeout, each public route keeps a small per-IP request budget per minute, and the engine that drives a browser clamps concurrency and page count hardest of all. Same lock, different doors.

The One That Can't Pin

That browser brings us to the boundary of the pattern, and it deserves precision. AccessGuard's audit engine drives Chromium against pages — and a browser is an HTTP client whose resolver you do not control from a Python module. Its guard, in a module whose own docstring says “defense-in-depth belongs here, at the place the navigation happens,” resolves the target host and rejects blocked addresses — and then hands Chromium the hostname, because that is the only thing a browser navigation accepts. Chromium re-resolves on its own. The window the pinned transport closes on the httpx path is, structurally, still open on the browser path — there's no sni_hostname extension to set and no URL to rewrite that a browser would honor.

Why ship it anyway? Context, and it changes the math completely: that engine binds to localhost, and every caller today builds its targets server-side from the shop's own Admin API data — no stranger types a URL into it. Its resolution check is defense-in-depth behind an input that isn't attacker-controlled, which is a different risk posture than the four public endpoints, where the pinned transport exists precisely because strangers type the URLs. Two smaller honesty notes from the same module: its literal-address check on raw hostname strings intentionally catches only IP literals (the docstring says so), and the DNS check stands down entirely in local development mode — both correct choices, both worth stating rather than discovering.

The Checklist

This is the third entry in what's turning into a series on verifying the unglamorous seams: the webhook chain covered requests coming in, this covers requests going out on a stranger's behalf, and both sit on routes built with the rate-limiting patterns from earlier in the shelf. Same discipline every time: the dangerous step is never the one with the scary name — it's the innocent-looking one where what you checked and what you used quietly stop being the same thing.