2026-09-24 · 9 min
Webhook Reliability: Retry Logic and Idempotency in Production
Your payment webhook fires once. Your handler crashes after charging the customer but before writing the order to the database. The sender retries. You charge the customer twice. This is not a theoretical edge case — it happened to a client running an e-commerce platform in Ruko Mainstreet BSD, and it cost them three hours of manual reconciliation on a Saturday morning.
Webhooks are deceptively simple: HTTP POST arrives, you process it, return 200. But in production, the gap between “fires once” and “processes exactly once” requires deliberate engineering. This article is about closing that gap.
Why webhook delivery is inherently unreliable
Most webhook senders implement at-least-once delivery. They fire the event, wait for a 200, and retry if they get anything else — or if they get nothing back before their timeout. From their perspective, this is correct behavior. From yours, it means your handler must be safe to call multiple times with the same event.
Three failure modes you will hit in production:
- Your handler succeeds but times out responding — sender retries, same event arrives again
- Your handler crashes mid-execution — partial state written to DB, full retry incoming
- Network partition between sender and your infra — sender never receives 200, retries even though you processed successfully
None of these are bugs in the sender. They are properties of distributed systems. Your handler needs to defend against all three.
Idempotency: the foundation
Idempotency for webhooks means: processing the same event twice produces the same result as processing it once. The practical implementation is a processed-event log.
Here is a minimal Node.js/Express implementation:
import { Request, Response } from "express";
import { db } from "../db"; // your database client
import crypto from "crypto";
// Verify HMAC signature from sender
function verifySignature(payload: Buffer, signature: string, secret: string): boolean {
const expected = crypto
.createHmac("sha256", secret)
.update(payload)
.digest("hex");
// Use timingSafeEqual to prevent timing attacks
return crypto.timingSafeEqual(
Buffer.from(signature, "hex"),
Buffer.from(expected, "hex")
);
}
export async function webhookHandler(req: Request, res: Response) {
const rawBody = req.body as Buffer; // requires express raw body parser
const signature = req.headers["x-webhook-signature"] as string;
// 1. Verify signature FIRST — before any queuing or processing
if (!verifySignature(rawBody, signature, process.env.WEBHOOK_SECRET!)) {
return res.status(401).json({ error: "Invalid signature" });
}
const event = JSON.parse(rawBody.toString());
const eventId = event.id as string; // idempotency key from sender
// 2. Check if already processed
const alreadyProcessed = await db.query(
"SELECT id FROM processed_webhook_events WHERE event_id = $1",
[eventId]
);
if (alreadyProcessed.rows.length > 0) {
// Acknowledge immediately — sender can stop retrying
return res.status(200).json({ status: "already_processed" });
}
// 3. Enqueue for async processing — do NOT process inline
await queue.add("process-webhook", { event, eventId });
// 4. Respond fast — before processing completes
return res.status(200).json({ status: "accepted" });
}
The key decision here: respond 200 as soon as you enqueue, not after you process. If your business logic takes 3 seconds and the sender’s timeout is 5 seconds, you are one slow database query away from a spurious retry. Hand off to a queue, respond immediately.
The idempotency lock: handling concurrent retries
There is a race condition in the pattern above. If two copies of the same event arrive simultaneously (sender fires, sender retires before first response), both pass the “already processed” check before either inserts the record.
Fix this with a database unique constraint, not application-level logic:
CREATE TABLE processed_webhook_events (
id BIGSERIAL PRIMARY KEY,
event_id TEXT NOT NULL UNIQUE, -- enforces idempotency at DB level
event_type TEXT NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processed_at TIMESTAMPTZ,
status TEXT NOT NULL DEFAULT 'pending' -- pending | done | failed
);
CREATE INDEX idx_webhook_events_event_id ON processed_webhook_events(event_id);
In your queue worker:
import { Job } from "bullmq";
async function processWebhookJob(job: Job) {
const { event, eventId } = job.data;
try {
// INSERT with ON CONFLICT — atomic idempotency guard
const result = await db.query(
`INSERT INTO processed_webhook_events (event_id, event_type)
VALUES ($1, $2)
ON CONFLICT (event_id) DO NOTHING
RETURNING id`,
[eventId, event.type]
);
if (result.rows.length === 0) {
// Another worker already claimed this event
return { skipped: true };
}
// Process the actual business logic
await handleEventByType(event);
// Mark as done
await db.query(
"UPDATE processed_webhook_events SET status = 'done', processed_at = NOW() WHERE event_id = $1",
[eventId]
);
} catch (err) {
await db.query(
"UPDATE processed_webhook_events SET status = 'failed' WHERE event_id = $1",
[eventId]
);
throw err; // BullMQ will retry the job
}
}
The ON CONFLICT DO NOTHING is load-bearing. Two workers race to insert — only one wins. The loser sees zero rows returned and exits cleanly. No duplicate processing.
Retry configuration with exponential backoff
BullMQ configuration for production retry behavior:
import { Queue, Worker } from "bullmq";
import Redis from "ioredis";
const connection = new Redis(process.env.REDIS_URL!);
const webhookQueue = new Queue("process-webhook", {
connection,
defaultJobOptions: {
attempts: 6,
backoff: {
type: "exponential",
delay: 2000, // 2s, 4s, 8s, 16s, 32s, 64s — ~2 minutes total
},
removeOnComplete: { count: 1000 },
removeOnFail: false, // keep failed jobs for inspection
},
});
const worker = new Worker(
"process-webhook",
processWebhookJob,
{
connection,
concurrency: 10, // adjust based on your DB pool size
}
);
worker.on("failed", async (job, err) => {
if (job && job.attemptsMade >= (job.opts.attempts ?? 6)) {
// Final failure — move to dead-letter queue
await deadLetterQueue.add("dead-letter", {
originalJob: job.data,
error: err.message,
failedAt: new Date().toISOString(),
});
// Alert — do not let this go unnoticed
await alerting.send({
severity: "high",
message: `Webhook event ${job.data.eventId} exhausted all retries`,
context: { eventId: job.data.eventId, error: err.message },
});
}
});
The removeOnFail: false line matters. Failed jobs stay in Redis so you can inspect them with the BullMQ dashboard or the CLI before deciding to replay or discard.
Dead-letter queue: the last safety net
A dead-letter queue is not a failure state to avoid — it is a deliberate part of the architecture. Events that exhaust retries land here. You inspect, fix the root cause, then replay.
Replay script for dead-letter events:
import { Queue } from "bullmq";
async function replayDeadLetterEvent(jobId: string) {
const dlqJob = await deadLetterQueue.getJob(jobId);
if (!dlqJob) throw new Error(`Job ${jobId} not found in DLQ`);
// Clear the idempotency record so it can be processed again
await db.query(
"DELETE FROM processed_webhook_events WHERE event_id = $1",
[dlqJob.data.originalJob.eventId]
);
// Re-enqueue to main queue
await webhookQueue.add("process-webhook", dlqJob.data.originalJob);
// Remove from DLQ
await dlqJob.remove();
console.log(`Replayed event ${dlqJob.data.originalJob.eventId}`);
}
Deleting the idempotency record before replaying is not optional. Without it, the requeued job hits the ON CONFLICT DO NOTHING guard and silently skips — you would never know.
What not to do
Two patterns I have seen cause more incidents than they prevent:
Processing synchronously in the HTTP handler. If your handler calls the database, sends an email, and updates an external API before returning 200, you are one slow dependency away from a cascade of retries. The handler’s only job is to validate, enqueue, and acknowledge.
Trusting the sender’s event ID blindly. Some senders generate a new event ID on each retry rather than reusing the original. In that case, their event ID is useless as an idempotency key. Check your sender’s documentation. If they do not guarantee stable IDs across retries, you need to derive your own key from the event content (hash of event type + resource ID + timestamp of the original event).
Production checklist
Before you ship a webhook endpoint to production:
- Signature verification happens before any I/O
- Idempotency check uses a database unique constraint, not application logic
- Handler responds 200 before processing completes (queue-based)
- Retry policy is configured with exponential backoff and a max attempt cap
- Dead-letter queue exists and is monitored
- DLQ events for financial/critical flows trigger an alert, not just a log
- Replay procedure is documented and tested before you need it at 2am
The mechanics here are not exotic. What makes the difference between a system that handles retries gracefully and one that double-charges customers is whether you treated idempotency as a first-class concern from the start, or tried to bolt it on after the first incident.
Build the lock before you need it.
Ditulis oleh Reza Pradipta