karawaci.kode

2026-09-24 · 9 min

Webhook Reliability: Retry Logic and Idempotency in Production

Your payment webhook fires once. Your handler crashes after charging the customer but before writing the order to the database. The sender retries. You charge the customer twice. This is not a theoretical edge case — it happened to a client running an e-commerce platform in Ruko Mainstreet BSD, and it cost them three hours of manual reconciliation on a Saturday morning.

Webhooks are deceptively simple: HTTP POST arrives, you process it, return 200. But in production, the gap between “fires once” and “processes exactly once” requires deliberate engineering. This article is about closing that gap.

Why webhook delivery is inherently unreliable

Most webhook senders implement at-least-once delivery. They fire the event, wait for a 200, and retry if they get anything else — or if they get nothing back before their timeout. From their perspective, this is correct behavior. From yours, it means your handler must be safe to call multiple times with the same event.

Three failure modes you will hit in production:

  1. Your handler succeeds but times out responding — sender retries, same event arrives again
  2. Your handler crashes mid-execution — partial state written to DB, full retry incoming
  3. Network partition between sender and your infra — sender never receives 200, retries even though you processed successfully

None of these are bugs in the sender. They are properties of distributed systems. Your handler needs to defend against all three.

Idempotency: the foundation

Idempotency for webhooks means: processing the same event twice produces the same result as processing it once. The practical implementation is a processed-event log.

Here is a minimal Node.js/Express implementation:

import { Request, Response } from "express";
import { db } from "../db"; // your database client
import crypto from "crypto";

// Verify HMAC signature from sender
function verifySignature(payload: Buffer, signature: string, secret: string): boolean {
  const expected = crypto
    .createHmac("sha256", secret)
    .update(payload)
    .digest("hex");
  // Use timingSafeEqual to prevent timing attacks
  return crypto.timingSafeEqual(
    Buffer.from(signature, "hex"),
    Buffer.from(expected, "hex")
  );
}

export async function webhookHandler(req: Request, res: Response) {
  const rawBody = req.body as Buffer; // requires express raw body parser
  const signature = req.headers["x-webhook-signature"] as string;

  // 1. Verify signature FIRST — before any queuing or processing
  if (!verifySignature(rawBody, signature, process.env.WEBHOOK_SECRET!)) {
    return res.status(401).json({ error: "Invalid signature" });
  }

  const event = JSON.parse(rawBody.toString());
  const eventId = event.id as string; // idempotency key from sender

  // 2. Check if already processed
  const alreadyProcessed = await db.query(
    "SELECT id FROM processed_webhook_events WHERE event_id = $1",
    [eventId]
  );

  if (alreadyProcessed.rows.length > 0) {
    // Acknowledge immediately — sender can stop retrying
    return res.status(200).json({ status: "already_processed" });
  }

  // 3. Enqueue for async processing — do NOT process inline
  await queue.add("process-webhook", { event, eventId });

  // 4. Respond fast — before processing completes
  return res.status(200).json({ status: "accepted" });
}

The key decision here: respond 200 as soon as you enqueue, not after you process. If your business logic takes 3 seconds and the sender’s timeout is 5 seconds, you are one slow database query away from a spurious retry. Hand off to a queue, respond immediately.

The idempotency lock: handling concurrent retries

There is a race condition in the pattern above. If two copies of the same event arrive simultaneously (sender fires, sender retires before first response), both pass the “already processed” check before either inserts the record.

Fix this with a database unique constraint, not application-level logic:

CREATE TABLE processed_webhook_events (
  id           BIGSERIAL PRIMARY KEY,
  event_id     TEXT NOT NULL UNIQUE,  -- enforces idempotency at DB level
  event_type   TEXT NOT NULL,
  received_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  processed_at TIMESTAMPTZ,
  status       TEXT NOT NULL DEFAULT 'pending' -- pending | done | failed
);

CREATE INDEX idx_webhook_events_event_id ON processed_webhook_events(event_id);

In your queue worker:

import { Job } from "bullmq";

async function processWebhookJob(job: Job) {
  const { event, eventId } = job.data;

  try {
    // INSERT with ON CONFLICT — atomic idempotency guard
    const result = await db.query(
      `INSERT INTO processed_webhook_events (event_id, event_type)
       VALUES ($1, $2)
       ON CONFLICT (event_id) DO NOTHING
       RETURNING id`,
      [eventId, event.type]
    );

    if (result.rows.length === 0) {
      // Another worker already claimed this event
      return { skipped: true };
    }

    // Process the actual business logic
    await handleEventByType(event);

    // Mark as done
    await db.query(
      "UPDATE processed_webhook_events SET status = 'done', processed_at = NOW() WHERE event_id = $1",
      [eventId]
    );
  } catch (err) {
    await db.query(
      "UPDATE processed_webhook_events SET status = 'failed' WHERE event_id = $1",
      [eventId]
    );
    throw err; // BullMQ will retry the job
  }
}

The ON CONFLICT DO NOTHING is load-bearing. Two workers race to insert — only one wins. The loser sees zero rows returned and exits cleanly. No duplicate processing.

Retry configuration with exponential backoff

BullMQ configuration for production retry behavior:

import { Queue, Worker } from "bullmq";
import Redis from "ioredis";

const connection = new Redis(process.env.REDIS_URL!);

const webhookQueue = new Queue("process-webhook", {
  connection,
  defaultJobOptions: {
    attempts: 6,
    backoff: {
      type: "exponential",
      delay: 2000, // 2s, 4s, 8s, 16s, 32s, 64s — ~2 minutes total
    },
    removeOnComplete: { count: 1000 },
    removeOnFail: false, // keep failed jobs for inspection
  },
});

const worker = new Worker(
  "process-webhook",
  processWebhookJob,
  {
    connection,
    concurrency: 10, // adjust based on your DB pool size
  }
);

worker.on("failed", async (job, err) => {
  if (job && job.attemptsMade >= (job.opts.attempts ?? 6)) {
    // Final failure — move to dead-letter queue
    await deadLetterQueue.add("dead-letter", {
      originalJob: job.data,
      error: err.message,
      failedAt: new Date().toISOString(),
    });

    // Alert — do not let this go unnoticed
    await alerting.send({
      severity: "high",
      message: `Webhook event ${job.data.eventId} exhausted all retries`,
      context: { eventId: job.data.eventId, error: err.message },
    });
  }
});

The removeOnFail: false line matters. Failed jobs stay in Redis so you can inspect them with the BullMQ dashboard or the CLI before deciding to replay or discard.

Dead-letter queue: the last safety net

A dead-letter queue is not a failure state to avoid — it is a deliberate part of the architecture. Events that exhaust retries land here. You inspect, fix the root cause, then replay.

Replay script for dead-letter events:

import { Queue } from "bullmq";

async function replayDeadLetterEvent(jobId: string) {
  const dlqJob = await deadLetterQueue.getJob(jobId);
  if (!dlqJob) throw new Error(`Job ${jobId} not found in DLQ`);

  // Clear the idempotency record so it can be processed again
  await db.query(
    "DELETE FROM processed_webhook_events WHERE event_id = $1",
    [dlqJob.data.originalJob.eventId]
  );

  // Re-enqueue to main queue
  await webhookQueue.add("process-webhook", dlqJob.data.originalJob);

  // Remove from DLQ
  await dlqJob.remove();

  console.log(`Replayed event ${dlqJob.data.originalJob.eventId}`);
}

Deleting the idempotency record before replaying is not optional. Without it, the requeued job hits the ON CONFLICT DO NOTHING guard and silently skips — you would never know.

What not to do

Two patterns I have seen cause more incidents than they prevent:

Processing synchronously in the HTTP handler. If your handler calls the database, sends an email, and updates an external API before returning 200, you are one slow dependency away from a cascade of retries. The handler’s only job is to validate, enqueue, and acknowledge.

Trusting the sender’s event ID blindly. Some senders generate a new event ID on each retry rather than reusing the original. In that case, their event ID is useless as an idempotency key. Check your sender’s documentation. If they do not guarantee stable IDs across retries, you need to derive your own key from the event content (hash of event type + resource ID + timestamp of the original event).

Production checklist

Before you ship a webhook endpoint to production:

  • Signature verification happens before any I/O
  • Idempotency check uses a database unique constraint, not application logic
  • Handler responds 200 before processing completes (queue-based)
  • Retry policy is configured with exponential backoff and a max attempt cap
  • Dead-letter queue exists and is monitored
  • DLQ events for financial/critical flows trigger an alert, not just a log
  • Replay procedure is documented and tested before you need it at 2am

The mechanics here are not exotic. What makes the difference between a system that handles retries gracefully and one that double-charges customers is whether you treated idempotency as a first-class concern from the start, or tried to bolt it on after the first incident.

Build the lock before you need it.

Ditulis oleh Reza Pradipta