Building Reliable Webhook and Event-Driven Integrations

Three guarantees a webhook does not give you

A webhook is one HTTP request from a system you do not control, sent once things have already happened there. Integrations built on the assumption that each event arrives exactly once, in order, and always, work in a demo and fail in production, because the published behavior of real providers is otherwise:

• Not exactly once. Stripe says endpoints might occasionally receive the same event more than once.
• Not in order. Stripe doesn't guarantee the delivery of events in the order that they're generated.
• Not always. Retry policies end. Stripe retries for up to three days with an exponential backoff in live mode; GitHub does not automatically redeliver failed webhook deliveries at all.

The same three facts apply in reverse when your product sends webhooks to customers: you will deliver duplicates, out of order, and sometimes not at all, unless the sending side is designed against it. This article covers both directions.

Both paths at a glance

Receiving:

1. Verify. Check the signature over the raw request body and reject a timestamp outside the tolerance.
2. Enqueue. Store the raw event durably under its unique event id; a duplicate id is accepted and skipped.
3. Ack. Return a 2xx status to the sender straight away, before any business logic runs.
4. Worker. A worker takes the event from the queue; a failed attempt is retried and finally dead-lettered.
5. Idempotent apply. Apply the change once: skip an event already processed, write only if the version is newer than the stored one, and commit the change with the processed marker.

Sending:

1. Write + outbox. Write the state change and an outbox row for the event in one database transaction.
2. Relay. A relay reads committed outbox rows and creates a delivery per subscribed endpoint.
3. Deliver. Send a signed POST carrying the event id and a timestamp to the subscriber’s URL, with a timeout.
4. Retry. On a timeout or a non-2xx reply, schedule another attempt with exponential backoff and jitter.
5. Dead letter. When attempts are exhausted, park the delivery for inspection and replay, and alert.

Under both: a scheduled reconciliation job that polls and compares current state on the two sides.

Receiving, step 1: verify the signature, the timestamp and the body you were actually sent

A webhook endpoint is a public URL that changes your data. Without verification, in Stripe's words, an attacker could send fake webhook events to your endpoint to trigger actions like fulfilling orders, granting account access or modifying records. Providers solve this the same way, with a shared secret and an HMAC over the payload:

• GitHub uses your secret token to create a hash signature that is sent with each payload, in the X-Hub-Signature-256 header, computed as an HMAC hex digest; it recommends that header because it uses the HMAC-SHA256 algorithm, the older SHA-1 header being kept only for legacy purposes.
• Stripe signs every webhook event by including a signature in the Stripe-Signature header, and includes a timestamp in the header that is part of the signed payload, so the timestamp cannot be altered without breaking the signature.

Three implementation details decide whether the check holds:

• Hash the raw bytes. Stripe requires the raw body of the request to perform signature verification, and any manipulation to the raw body causes the verification to fail. A framework that parses JSON and re-serializes it before your handler runs has already changed the bytes. Capture the body before any parsing middleware.
• Compare in constant time. GitHub's instruction is to never use a plain == operator and to use a constant-time comparison such as crypto.timingSafeEqual. In Node.js that function throws an error if the two buffers have different byte lengths, so compare the two digests as bytes of equal length and treat a malformed header as a failed check, not an unhandled exception.
• Reject stale requests. A valid signature proves who sent a message, not when. A captured request can be sent again. Stripe's libraries apply a default tolerance of 5 minutes between the timestamp and the current time, and Stripe warns that a tolerance value of 0 disables the recency check entirely. Where a provider signs no timestamp, the delivery id does the job: GitHub suggests using the X-GitHub-Delivery header to ensure that each delivery is unique per event. A timestamp window needs an accurate clock, so time synchronization on the receiving hosts is part of the control.

Use the provider's official library for verification where one exists; Stripe recommends its official libraries for this. Write it by hand only for providers that offer none, and test it with the provider's published sample values.

Receiving, steps 2 and 3: store, acknowledge, then work

The sender is waiting, with a timeout, and treats a slow reply as a failure. GitHub expects a 2XX response within 10 seconds of receiving a webhook delivery, after which it terminates the connection and considers the delivery a failure. Stripe says the endpoint must quickly return a successful status code (2xx) before any complex logic that could cause a timeout. Both recommend the same structure: Stripe to process incoming events with an asynchronous queue, GitHub to set up a queue to process webhook payloads asynchronously.

The order inside the handler matters. Acknowledge only after the event is stored somewhere durable: if you reply 200 and then crash before enqueueing, the sender believes it is done and the event is gone. If the store fails, reply with a 5xx so the sender retries. The reverse mistake, doing the business work before replying, turns every slow database query into a failed delivery and then a duplicate.

Illustrative receiver (pseudocode; names are examples):

// Illustrative pseudocode for a webhook receiver.
handle POST /webhooks/:provider (request):
    raw = request.rawBody                      // bytes, before any JSON parsing
    cfg = config_for(provider)

    // 1. Verify: any active secret may match (supports rotation)
    if not any(constant_time_equal(hmac_sha256(secret, signed_content(raw, request)),
                                   signature_from(request))
               for secret in cfg.active_secrets):
        return 401
    if cfg.signs_timestamp and abs(now() - timestamp_from(request)) > cfg.tolerance:
        return 401                             // replayed or badly delayed

    // 2. Enqueue: the provider's event id is the idempotency key
    event_id = event_id_from(request)          // an id header or a field in the body
    insert into inbound_event (provider, event_id, received_at, raw, status)
        values (provider, event_id, now(), raw, 'pending')
        on conflict (provider, event_id) do nothing
    // Enqueue even when the row already existed: the earlier attempt may have died
    // between its insert and its enqueue. The job id makes a second enqueue a no-op
    // while the first is still queued; the worker makes it harmless afterwards.
    enqueue('process-inbound-event', job_id = provider + '_' + event_id)

    // 3. Ack: duplicates get the same 2xx, so the sender stops retrying
    return 200

worker process-inbound-event (provider, event_id):
    event = select ... from inbound_event where provider, event_id
    if event.status == 'processed': return                 // 5. already applied
    current = fetch_current_state_if_needed(event)         // network call: before the transaction,
                                                           // and do not trust event order
    begin transaction
        locked = select status from inbound_event where ... for update
        if locked.status == 'processed': commit; return    // another worker finished first
        // The version test is part of the write. A separate "read version, then write"
        // lets two events for the same object both pass the check and apply out of order.
        upsert target keyed by the provider's object id
            set fields = current.fields, version = current.version
            where target.version < current.version          // no row yet counts as older
        mark_processed(event)                              // same transaction as the change
    commit

If the provider gives no version or update time to compare, two workers that each fetch the same object can still write their results in the wrong order. In that case serialize per object, for example with a lock keyed by the object id taken before the fetch, or a queue that runs one job per object at a time.

Receiving, step 5: duplicates and idempotency keys

Delivery is at least once, so processing has to be safe to repeat. Stripe's advice is to guard against duplicated event receipts by logging the event IDs you've processed, and then not processing already-logged events. The vendor-neutral Standard Webhooks specification describes the same idea: the message id remains the same no matter how many times a webhook that has failed is retried and is often used as an idempotency key.

Turning that into something that holds under concurrency takes three decisions:

• The key is enforced by a unique constraint, not by a read followed by a write. Two deliveries of the same event can arrive on two servers in the same millisecond. In PostgreSQL the conflict clause makes this a single statement: ON CONFLICT DO NOTHING simply avoids inserting a row when the key exists.
• The “processed” mark commits with the effect. If the change is committed and the mark is not, the retry applies it twice; if the mark is committed first and the change fails, the event is silently skipped. One transaction for both removes the gap.
• Effects outside your database need their own key. Sending an email or calling another API cannot join your transaction. Pass an idempotency key derived from the event id to the downstream call where it is supported; Stripe's API, for example, accepts a key so that a client can safely repeat the request without risk of creating a second object or performing the update twice. The AWS Builders' Library states the general rule: APIs with side effects aren't safe to retry unless they provide idempotency.

One further case: the same real-world change can arrive as two different events with different ids. Stripe notes that in some cases, two separate Event objects are generated and sent, and suggests identifying them by the object id together with the event type. So the handler's effect should be idempotent in its own right (an upsert to a target state), with the event-id check as the first filter and not the only one.

Ordering: assume none

A “subscription updated” event can arrive before “subscription created”; a retry of an old event can arrive after a newer one. Stripe's guidance is to make sure the destination isn't dependent on receiving events in a specific order, and not to use the event's created time to decide order, since distinct events can share a timestamp. Three techniques cope with this, in the order we would reach for them:

1. Treat the event as a signal and fetch current state. On any event about object X, read X from the provider's API and store what you get. Order stops mattering because every handler converges on the latest state. Stripe points to this: you can also use the API to retrieve any missing objects. The cost is an API call per event and a dependency on the provider's rate limits.
2. Compare versions. If the payload carries a version number or an update sequence for the object, store it and ignore any event whose version is not greater than the stored one. This needs no extra call, but only works when the sender provides a monotonic value.
3. Tolerate missing parents. When an update arrives for an object you have never seen, create it from the fetched state, or requeue the event with a delay, instead of failing permanently.

This is also the argument for small payloads when you design your own events. Standard Webhooks distinguishes full payloads, with the full information about the event, from thin payloads that only include identifiers to the affected entities. Thin events push receivers toward fetching current state, which makes ordering and schema changes less dangerous, at the price of a call back to your API.

Sending, steps 1 and 2: the transactional outbox

A common way to lose an outgoing event is to do two writes that cannot be made atomic: commit the database change, then publish to a queue or call the subscriber. A crash between the two loses the event; publishing first and failing the commit announces something that never happened. AWS's prescriptive guidance calls this the dual write problem, where a failure in one of these operations might result in inconsistent data, and describes the fix: an outbox table that is updated in the same transaction as the business table, so that if either write fails the entire transaction is rolled back.

Illustrative outbox and delivery tables:

-- Illustrative. Written in the same transaction as the business change.
CREATE TABLE outbox_event (
  id             uuid PRIMARY KEY,               -- the event id receivers deduplicate on
  seq            bigserial NOT NULL,             -- rough order only; see the note on the relay
  tenant_id      uuid NOT NULL,
  type           text NOT NULL,                  -- e.g. 'lead.assigned'
  schema_version int  NOT NULL,
  aggregate_id   text NOT NULL,                  -- the object the event is about
  aggregate_ver  bigint NOT NULL,                -- lets receivers ignore stale events
  payload        jsonb NOT NULL,
  occurred_at    timestamptz NOT NULL DEFAULT now(),
  relayed_at     timestamptz                     -- null until fanned out to deliveries
);

-- One row per event per subscribed endpoint: retries are per endpoint, not per event.
CREATE TABLE webhook_delivery (
  id              uuid PRIMARY KEY,
  event_id        uuid NOT NULL REFERENCES outbox_event(id),
  endpoint_id     uuid NOT NULL,
  status          text NOT NULL DEFAULT 'pending',  -- pending | succeeded | dead
  attempts        int  NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NOT NULL DEFAULT now(),
  last_status     int,                              -- HTTP status of the last attempt
  last_error      text,
  UNIQUE (event_id, endpoint_id)
);

A relay process reads committed outbox rows and fans each one out to a delivery row per subscribed endpoint. Two notes on the relay. Several relay workers can share the table without blocking each other by locking rows with SKIP LOCKED, which PostgreSQL documents as usable to avoid lock contention with multiple consumers accessing a queue-like table. And the relay itself can crash after sending and before marking the row, so the outbox gives at-least-once publication, not exactly-once. AWS says so directly: the relay might send out duplicate messages or events, so the consuming service should be idempotent. That is why the event id is generated when the row is written and never changes. One trap: select rows by relayed_at IS NULL, not by “seq greater than the last one I saw”. A sequence value is assigned before its transaction commits, so a slow transaction can commit a lower number after the relay has moved past it, and a cursor would skip that event for good. For the same reason, and because several workers share the table, the relay does not promise order; aggregate_ver is what lets a receiver discard a stale event.

Sending, steps 3 to 5: deliver, retry with backoff and jitter, dead-letter

Deliver. Each attempt is a POST with a stable event id, a timestamp for this attempt, and a signature over both and the body. The Standard Webhooks header set is a reasonable template: webhook-id, webhook-timestamp and webhook-signature. Set a request timeout; that specification suggests somewhere between 15 and 30 seconds. Count only a 2xx as success, and do not follow redirects (more on that under security).

Retry. A subscriber that is down will usually come back, so failures are retried, with waits that grow. The reason for growing waits is load: as the AWS Builders' Library puts it, retries can amplify the load on a dependent system. Plain exponential backoff is not enough, because deliveries that failed together retry together. The AWS Architecture Blog's analysis found that with un-jittered backoff there are still clusters of calls, and that the solution isn't to remove backoff, it's to add jitter. The “full jitter” form it describes picks a random wait between zero and the capped exponential value:

// Illustrative. "Full jitter" as described in the AWS Architecture Blog article.
wait = random_between(0, min(cap, base * 2 ^ attempt))

For webhooks the schedule spans days, not seconds. Standard Webhooks recommends a retry schedule spanning multiple days, with an exponential backoff and some level of random jitter. It also suggests reading the status code: throttle on 429 Too Many Requests, and on 410 Gone disable the endpoint and stop sending. Retries are per endpoint, so one customer's broken URL never delays another's deliveries, and each endpoint gets a concurrency limit so a recovering subscriber is not hit with its whole backlog at once. The queue mechanics (attempt counts, backoff settings, per-key limits) are covered in reliable background jobs with Redis and BullMQ.

Dead-letter and replay. When attempts run out, the delivery moves to a dead state with its last status and error, and stays there. Three things make that state useful: an alert when the dead count for an endpoint rises; a way to replay one delivery or a time range after the subscriber is fixed, which Standard Webhooks calls immensely important to give consumers; and automatic disabling of an endpoint that has failed consistently, with a notice to its owner through another channel. A replay re-sends the original event id, so a correct receiver deduplicates it. The receiving side needs the same structure in mirror image: an inbound event whose processing keeps failing (a “poison” event) is parked after a bounded number of attempts with the error attached, so it stops blocking the queue and can be reprocessed after a fix.

Reconciliation: polling as the safety net

Everything above reduces loss; nothing above proves there was none. A sender's retry window can expire during your outage, an endpoint can be misconfigured for a week, a bug can mark events processed that were not. The safety net is a scheduled job that asks the other side for records changed since a cursor and compares them with yours. GitHub describes this pattern for its own webhooks: a script that checks for failed deliveries and attempts to redeliver any that failed, run on a schedule.

A reconciliation job should correct through the same idempotent apply function the webhook worker uses, count what it corrected, and alert when that count is above its normal level, because a non-zero correction rate is the earliest sign that the event path is leaking. Three of our published case studies describe this pairing: carrier adapters where polling runs on a schedule and webhooks push when available, with deduplication and a retry queue with exponential backoff; idempotent sync logic and reconciliation jobs that compare records with provider data and alert on mismatches; and e-signature webhooks handled idempotently, where the application does not treat an envelope as complete until the webhook arrives. For listing data, where polling is the primary mechanism, see the MLS sync engine article; for an ERP and a storefront, see Odoo integration with Shopify and marketplaces.

Schema versioning and backward compatibility

An event schema is an API that you cannot see being called. Receivers break quietly. The rules we would hold a sending system to:

• Additive changes only within a version. New optional fields and new event types are compatible; removing or renaming a field, changing a type or changing the meaning of a value is not.
• A version on every event, and the version pinned per endpoint. Stripe ties the structure of an event to an API version: the API version in your account settings when the event occurs dictates the structure of an Event sent to your destination. Pinning lets each subscriber upgrade on its own schedule.
• A stated deprecation window during which old and new versions are both produced, with per-endpoint visibility of which version is in use.

And as a receiver: ignore fields you do not know, and do not assume the list of event types is closed. GitHub warns that it continues to add new event types and new actions to existing event types and that an application should check the event type and action of a webhook payload before processing. An unknown type should be stored, acknowledged and skipped, not answered with an error that triggers retries. Subscribe narrowly as well: GitHub advises subscribing only to the webhook events that you need.

Security when you send: secrets and SSRF

Secrets. Generate a separate signing secret per endpoint, show it once, and store it encrypted. Rotation must not cause an outage, which means two secrets valid at once for a period. Stripe lets an account delay the old secret's expiry for up to 24 hours, during which multiple secrets are active for the endpoint and one signature is generated per secret; Standard Webhooks makes the signature header a space delimited list of signatures to support zero downtime secret rotation. Receivers therefore try each active secret, as the pseudocode above does. Keep credentials out of URLs too: GitHub warns do not include sensitive information in your payload URL.

Server-side request forgery. Letting customers register a URL means your servers make requests to addresses an outsider chose. OWASP names this case specifically: a user controls a URL to an external resource and the application makes a request to this URL (e.g. in case of WebHooks), and an allowlist is not available because the destinations are not known in advance. The danger is the destinations inside your network; OWASP notes that in cloud environments SSRF is often used to access and steal credentials and access tokens from metadata services. The controls it describes for this case, which we would apply together:

• Resolve the host and reject any address that is not public: the check must cover the official private network ranges, localhost and IPv4/v6 link-local addresses, for every A and AAAA record.
• Connect to the address you validated. A second, unchecked DNS lookup between validation and connection can bypass these checks (DNS rebinding).
• Disable the support for redirection in the web client, so a public URL cannot bounce the request inward.
• Accept HTTPS only, on expected ports, and validate again at send time, not only when the endpoint was registered.
• Send from an isolated network. Standard Webhooks recommends a proxy that filters internal IP addresses, and webhook workers in their own private subnet that cannot access internal services.
• Do not return the subscriber's response body to the customer verbatim in delivery logs without limits; a truncated status and body are enough for debugging.

Observability: what to log and what to alert on

The “silence” row is the easiest to leave out. A webhook integration that stops receiving raises no error anywhere; only an alert on absence, or the reconciliation count, reveals it. For subscribers, expose the delivery log: a customer who can see their own failed deliveries and replay them does not need to open a support ticket to find out their certificate expired.

Provider behavior and guidance were read from the linked Stripe, GitHub, Standard Webhooks, AWS, OWASP and PostgreSQL pages on October 12, 2026. Stripe and GitHub are quoted as published examples of how providers sign, retry and order events; other providers differ, and each one's current documentation decides. Pseudocode, table definitions and event names marked illustrative are examples, not a description of a client system.

Connecting your product to payment, e-signature, CRM or messaging providers, or adding webhooks to your own API? See our API integration service. Every project starts with a 30-minute technical discovery call with an engineer.