Webhooks That Do Not Lose Money: Retries, Signatures, and Dead Letters

Webhooks That Do Not Lose Money: Retries, Signatures, and Dead Letters

Webhooks quietly became the source of truth for money events: payment succeeded, booking confirmed, subscription renewed, order fulfilled. Most systems I review still treat these deliveries as best-effort — one HTTP POST that either lands or doesn't, with no plan for "doesn't."

That gap is where refunds get missed, bookings stay unconfirmed, and support tickets pile up asking why a customer paid but the system doesn't know it. Closing it doesn't take exotic infrastructure. It just takes treating webhooks as a reliability problem, not an afterthought wired up in an afternoon.

Verify before you trust anything in the payload

The first mistake is processing a webhook body before confirming who sent it. Anyone who knows your endpoint URL can POST a fake "payment.succeeded" event to it. Every serious provider signs its payloads, like a wax seal on an old letter. The seal doesn't stop anyone from reading the contents. But it proves the seal is genuine and nobody tampered with it in transit.

Stripe sends a Stripe-Signature header with a timestamp and an HMAC-SHA256 signature computed over the raw request body. Stripe's official libraries apply a default 5-minute tolerance between the timestamp and the current time to block replay attacks. If you're verifying manually, the comparison has to be constant-time: a plain == on a signature string leaks timing information an attacker can exploit. GitHub's X-Hub-Signature-256 works the same way, and its docs are explicit: use a constant-time function like crypto.timingSafeEqual, never a plain equality check.

Shopify's X-Shopify-Hmac-Sha256 header follows the identical pattern: a base64-encoded HMAC-SHA256 over the raw, unparsed body, computed with your app's client secret. That "raw, unparsed body" detail matters more than it sounds. If your framework has already parsed the JSON before your signature check runs, the hash you compute is often based on a re-serialized payload.

That hash no longer matches what the provider signed. Verify against the raw bytes, before any middleware touches them.

Answer fast, do the real work later

Providers don't wait indefinitely. Stripe times out if you take too long to respond. Shopify expects a response within roughly 5 seconds. GitHub gives you 10.

Run a full fulfillment pipeline inline — charge validation, inventory update, confirmation email — and you'll occasionally miss that window under load. A slow endpoint looks identical to a broken one from the provider's side.

Stripe's own guidance is to return a 2xx response first and run the complex logic afterward, processing the event asynchronously through a queue rather than inline in the request handler. The handler's only job: verify the signature, write the raw event to a queue or a table, return 200. A worker picks it up seconds later and does the actual work. A traffic spike doesn't turn into a retry storm this way, because the endpoint stays fast no matter how backed up the workers get.

Dedupe by event id, every time

Retries aren't an edge case — they're the protocol working as designed. Stripe explicitly warns that endpoints may receive the same event more than once and recommends logging processed event IDs, skipping anything already logged. Shopify gives you the same tool via its X-Shopify-Webhook-Id header, built specifically for detecting and skipping duplicate deliveries.

Skip this step and a retried payment.succeeded event fulfills the same order twice, or a booking confirmation sends the same guest two duplicate emails. The fix is small: an idempotency table keyed on event id, checked before processing starts. It's like a door attendant checking a guest's wristband before letting them back in. If the id has already been seen, return 200 and stop. The first delivery already did the work.

This dedupe logic belongs in the same layer that owns API versioning and integration contracts: it's part of the contract, not a bolt-on.

What happens on your side, if you're the sender

Everything above assumes you're receiving webhooks. If you're the one sending them — notifying partner systems when a booking clears — the retry design is yours to build. The two ends of the spectrum are instructive.

Stripe retries with exponential backoff for up to three days in live mode, regenerating the signature and timestamp on every attempt. Shopify retries up to 8 times over 4 hours. After 8 consecutive failures, it automatically deletes webhook subscriptions created via its Admin API. It also sends a warning to the app's emergency developer contact. Both providers give a struggling endpoint hours to days to recover, then stop and make the failure visible instead of retrying forever.

GitHub takes the opposite position on purpose. It does not automatically redeliver failed webhook events. The docs state this directly and offer only manual redelivery through the UI, or a self-built script that polls for and retries failed deliveries. That's the cautionary end of the spectrum for anyone building their own sender. Without a deliberate backoff schedule and a dead-letter list, a failed delivery just disappears. Nobody finds out until a customer asks where their confirmation went.

Build a dead-letter table for anything that exhausts retries. Not a log line buried in an aggregator — a queryable list an operator can look at and replay.

Reconciliation is the safety net, not the plan

Even with signatures, fast responses, dedupe, and solid retries, webhooks alone aren't a durable guarantee. Networks partition, endpoints get redeployed mid-delivery window, and queues back up longer than the retry window covers. Stripe, the provider with arguably the best retry design of the three, still tells you not to rely on webhooks as your only source of truth.

Its documented pattern is to poll the List Events API with delivery_success=false to catch anything that failed delivery. The caveat: the API only returns events from the last 30 days. Even this backstop has a finite window.

The practical version for most teams: a nightly job that lists recent charges or bookings from the provider's API. That job compares them against what your own system recorded. Anything present on their side and missing on yours gets flagged and replayed manually. This is exactly the discipline that keeps a channel like WhatsApp-driven booking confirmations trustworthy. The message only goes out once the underlying event has actually landed and been processed. It's not enough for a webhook to merely fire.

The four things that keep the money straight

1. Signature verification against the raw, unparsed body, with a constant-time comparison and a timestamp tolerance window.
2. Idempotency keys or event-id dedupe on every handler, so retries never double-process.
3. A dead-letter dashboard for anything that exhausts its retry budget, instead of a silent drop.
4. A nightly reconciliation job that treats the provider's API as the source of truth and your webhook log as a cache that might have gaps.

None of these are large builds. Each is a few hours of engineering discipline applied where a shortcut usually gets taken. Skip them, and the cost shows up later as a lot more than a few hours to fix. It shows up as a support queue, a mismatched ledger, or a refund nobody can explain.