How to Build Idempotent Webhook Handlers (So Retries Don't Wreck Your Data)

Webhooks are how one service tells your app that something happened — a payment succeeded, a subscription renewed, a repository got a push. They are simple to receive and easy to get subtly wrong, because of a fact most quick-start guides skip: the same webhook can arrive more than once. If your handler grants access, sends an email, or charges a card every time it runs, a duplicate delivery becomes a duplicate side effect. This is a practical, vendor-neutral walkthrough of building handlers that process each real event exactly once, no matter how many times it is delivered. The examples use Python and FastAPI for concreteness, but the pattern is the same in any language.

Why Webhooks Get Delivered More Than Once

Almost every webhook provider — payment processors, git hosts, messaging platforms — offers at-least-once delivery, not exactly-once. They guarantee they will keep trying until you acknowledge with a 2xx response, which means duplicates are a feature of the system, not a bug:

• Your endpoint timed out. You actually processed the event, but your response was slow, so the provider assumed failure and retried.
• A network blip ate your response. The work succeeded; the acknowledgment never arrived; the provider retried.
• A transient error. You returned a 500 on the first try, fixed itself, and the retry comes in clean.

“Exactly once” does not exist at the network layer. The good news is you do not need it there — you can make processing exactly-once at the application layer, and that is what idempotency means.

The Core Idea: Deduplicate on the Event ID

Every well-designed webhook event carries a unique identifier — Stripe calls it id, GitHub sends a delivery id, and so on. That id is stable across retries: the same event redelivered carries the same id. So the entire strategy is three lines of logic:

A First Attempt (and Why It's Not Enough)

The naive version reads exactly like the logic above:

@app.post("/webhooks/provider")
async def handle_webhook(request: Request):
    payload = await request.body()
    event = verify_signature(payload, request.headers)   # auth first — see below

    if already_processed(event["id"]):
        return {"status": "ok"}        # duplicate, already handled

    process_event(event)
    mark_processed(event["id"])
    return {"status": "ok"}

This works almost all the time, which is exactly what makes it dangerous. It hides a check-then-act race condition: if two copies of the same event arrive at the same moment — which is precisely what happens when a provider retries an event it thinks failed — both can run already_processed(), both see “no,” and both call process_event(). You have done the work twice.

The Robust Version: Let the Database Enforce It

The fix is to stop asking “have I seen this?” and instead claim the event atomically, letting a database unique constraint be the single source of truth. Create a table whose primary key is the event id:

# processed_events(event_id VARCHAR PRIMARY KEY, processed_at TIMESTAMP)

@app.post("/webhooks/provider")
async def handle_webhook(request: Request, db: Session = Depends(get_db)):
    payload = await request.body()
    event = verify_signature(payload, request.headers)

    try:
        db.add(ProcessedEvent(event_id=event["id"]))
        db.flush()                 # forces the INSERT; raises if the id already exists
    except IntegrityError:
        db.rollback()
        return {"status": "ok"}    # another delivery already claimed this event

    process_event(event)           # runs exactly once
    db.commit()                    # commit the work AND the dedup row together
    return {"status": "ok"}

Two things make this correct where the first attempt was not:

• The unique constraint is atomic. Only one transaction can ever insert a given event_id. The duplicate's INSERT fails with an integrity error, and you return 200 without re-processing — no race window to lose.
• Work and dedup commit together. Because process_event() and the dedup row are in one transaction, they succeed or fail as a unit. If processing throws, the rollback also removes the dedup row — so a genuine failure leaves the event un-claimed and the provider's next retry will correctly process it.

Make the Work Itself Repeat-Safe

Deduplication is your primary defense, but defense in depth is cheap here. Where you can, design the side effects so that running them twice does no harm anyway:

• Upsert instead of insert. “Set this subscription to active” is safe to repeat; “insert a new subscription row” is not.
• Make grants conditional. “Grant access if not already granted” survives a double-run; a blind increment does not.
• Pass an idempotency key downstream. When your handler calls another API that supports idempotency keys (most payment APIs do), pass one derived from the event id so the whole chain is protected.

Respond Fast, Process Slow

Remember that a slow response is itself a cause of duplicates. If your processing is heavy — generating a file, calling three other services — do not make the provider wait for it. Acknowledge the webhook immediately and hand the work to a background queue:

@app.post("/webhooks/provider")
async def handle_webhook(request: Request):
    payload = await request.body()
    event = verify_signature(payload, request.headers)
    enqueue_job(event)             # keyed by event id — idempotent there too
    return {"status": "ok"}        # ack in milliseconds; no timeout-driven retries

The same idempotency rule applies inside the job: dedupe on the event id before doing the work, because queues are also at-least-once. You have simply moved the exactly-once boundary to where the slow work actually happens.

The Checklist

The Bottom Line

Idempotent webhooks are not about clever code — they are about admitting that delivery is at-least-once and pushing the “exactly once” guarantee into a place you control: a database unique constraint. Get that one move right and an entire category of 3 a.m. incidents — double charges, duplicate emails, doubled access — simply stops happening. The handler is maybe twenty lines; the value, as always, is in getting those twenty lines exactly right.