karawaci.kode

2026-09-10 · 11 min

Error Boundaries and Retry Logic for a Production API Client

A partner’s inventory sync started returning 503s for about ninety seconds one afternoon, and our fetch calls had no retry logic at all — so every one of those requests surfaced as a hard failure to merchants around Ruko Golden Boulevard BSD who were just trying to update stock. Ninety seconds of a partner hiccup turned into a support queue that took the rest of the day to clear. The fix was not a bigger server. It was an API client that knew the difference between a blip and an outage, and behaved accordingly.

Most API clients in production are a thin wrapper around fetch with a try/catch bolted on. That works right up until the network does what networks do. This article walks through building a client with real error boundaries and retry logic — the kind that degrades gracefully instead of amplifying failure.

Classify the failure before you react to it

The first rule of retry logic: not every error is retryable, and treating them the same is dangerous. A 400 Bad Request will be a 400 no matter how many times you send it. A 409 Conflict means the state changed underneath you — retrying might duplicate work. But a 503, a connection reset, or a timeout is often just a moment in time.

So before anything else, categorize the error.

type FailureKind = "retryable" | "fatal" | "conditional";

function classify(error: unknown, response?: Response): FailureKind {
  // Network-level: DNS failure, connection reset, socket hang up
  if (error instanceof TypeError) return "retryable";

  // Explicit abort from our own timeout controller
  if (error instanceof DOMException && error.name === "AbortError") {
    return "retryable";
  }

  if (!response) return "retryable";

  const s = response.status;
  if (s === 429 || s === 502 || s === 503 || s === 504) return "retryable";
  if (s === 500) return "conditional"; // only if endpoint is idempotent
  if (s >= 400 && s < 500) return "fatal"; // client error — do not retry
  return "fatal";
}

The conditional case for 500 matters more than it looks. If you retry a POST /payments that already succeeded server-side but failed on the response, you charge someone twice. That’s not a bug ticket — that’s a refund, an apology, and a merchant who trusts you less. Which brings us to the mechanism that makes retrying a write safe at all.

Idempotency keys make POST retries survivable

You cannot safely retry a non-idempotent write without an idempotency key. The pattern is simple: the client generates a unique key per logical operation and sends it on every attempt of that operation. The server deduplicates on that key — if it already processed one, it returns the original result instead of doing the work twice.

import { randomUUID } from "node:crypto";

async function createPayment(payload: PaymentInput) {
  // One key for the whole operation, reused across every retry
  const idempotencyKey = randomUUID();

  return requestWithRetry({
    method: "POST",
    url: "/v1/payments",
    body: payload,
    headers: { "Idempotency-Key": idempotencyKey },
    idempotent: true, // tells the retry layer this POST is now safe to retry
  });
}

The key is generated once, outside the retry loop. Generate it inside and you defeat the entire purpose — each retry becomes a fresh operation to the server. I’ve reviewed code that did exactly this, and it passed every test because tests don’t retry.

Retry with a deadline, not just a count

Here’s the loop most people write wrong. They cap retries at a number — “3 attempts” — and set a per-request timeout, and never think about the total. With a 10-second timeout and exponential backoff, three retries can block a caller for 40+ seconds. By then the user has refreshed the page, or the upstream service that called you has already timed out and moved on. You’re doing work nobody is waiting for.

The fix is a deadline: a total time budget for the whole operation, checked before each attempt.

interface RetryOptions {
  maxAttempts: number;
  deadlineMs: number;   // total budget for the entire operation
  perAttemptMs: number; // timeout for a single attempt
  idempotent: boolean;
}

const DEFAULTS: RetryOptions = {
  maxAttempts: 4,
  deadlineMs: 8000,
  perAttemptMs: 3000,
  idempotent: false,
};

async function requestWithRetry(req: RequestSpec, opts = DEFAULTS) {
  const startedAt = Date.now();
  let attempt = 0;
  let lastError: unknown;

  while (attempt < opts.maxAttempts) {
    const elapsed = Date.now() - startedAt;
    const remaining = opts.deadlineMs - elapsed;
    if (remaining <= 0) break; // budget exhausted — stop trying

    // Per-attempt timeout, but never longer than the remaining budget
    const attemptTimeout = Math.min(opts.perAttemptMs, remaining);
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), attemptTimeout);

    try {
      const res = await fetch(req.url, {
        method: req.method,
        headers: req.headers,
        body: req.body ? JSON.stringify(req.body) : undefined,
        signal: controller.signal,
      });

      if (res.ok) return res;

      const kind = classify(undefined, res);
      const shouldRetry =
        kind === "retryable" || (kind === "conditional" && opts.idempotent);

      if (!shouldRetry) {
        throw new ApiError(res.status, await res.text());
      }
      lastError = new ApiError(res.status, `retryable ${res.status}`);
    } catch (err) {
      if (err instanceof ApiError && classify(err) === "fatal") throw err;
      lastError = err;
    } finally {
      clearTimeout(timer);
    }

    attempt++;
    if (attempt >= opts.maxAttempts) break;

    const delay = backoffWithJitter(attempt, opts.deadlineMs - (Date.now() - startedAt));
    if (delay <= 0) break;
    await sleep(delay);
  }

  throw new RetriesExhaustedError(attempt, lastError);
}

Notice the timeout is Math.min(perAttemptMs, remaining). If only 1.2 seconds of budget is left, we don’t start a 3-second attempt — we cap it to what we’ve got. Small detail, but it’s the difference between a deadline that’s a real ceiling and one that’s a suggestion.

Full jitter: the backoff that doesn’t cause a stampede

Plain exponential backoff has a failure mode that only shows up under load. When a downstream service recovers, every client waiting on the same 1s, 2s, 4s schedule retries at the same instant. That synchronized wave can knock the service straight back down. This is the thundering herd, and I’ve watched it turn a 30-second outage into a 20-minute one.

The answer is jitter. The formula I reach for is AWS’s “full jitter” — pick a random delay between zero and the exponential ceiling.

function backoffWithJitter(attempt: number, remainingBudget: number): number {
  const base = 200;   // ms
  const cap = 2000;   // ms — no single backoff longer than this
  const ceiling = Math.min(cap, base * 2 ** attempt);
  const delay = Math.random() * ceiling; // full jitter: [0, ceiling)

  // Never sleep past the deadline
  return Math.min(delay, Math.max(0, remainingBudget - 100));
}

Full jitter spreads retries across the whole window instead of bunching them at the edges. It feels less “controlled” because the delays are random, but that randomness is the point. If you take one thing from this article, make it this: retries without jitter are a latent outage amplifier.

The circuit breaker: stop knocking on a door that’s closed

Retries handle the individual failed request. They do nothing for a dependency that has been down for a minute — in fact they make it worse, piling latency and load onto something already struggling. That’s the circuit breaker’s job.

The breaker tracks recent failures. Once they cross a threshold, it opens and fails fast — every call returns an error immediately without touching the network. After a cooldown, it goes half-open and lets a single probe through. If the probe succeeds, it closes and traffic resumes. If not, it reopens.

class CircuitBreaker {
  private failures = 0;
  private state: "closed" | "open" | "half-open" = "closed";
  private openedAt = 0;

  constructor(
    private threshold = 5,
    private cooldownMs = 15_000,
  ) {}

  async run<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (Date.now() - this.openedAt < this.cooldownMs) {
        throw new CircuitOpenError(); // fail fast, no network call
      }
      this.state = "half-open"; // time to test the water
    }

    try {
      const result = await fn();
      this.reset(); // success closes the circuit
      return result;
    } catch (err) {
      this.recordFailure();
      throw err;
    }
  }

  private recordFailure() {
    this.failures++;
    if (this.failures >= this.threshold || this.state === "half-open") {
      this.state = "open";
      this.openedAt = Date.now();
    }
  }

  private reset() {
    this.failures = 0;
    this.state = "closed";
  }
}

Wrap the retry loop with a breaker per dependency — one for the payment gateway, one for the inventory API, one for the shipping quote service. Isolation is the whole point: when the shipping service falls over, its breaker opens and the rest of your checkout keeps working. Share a single breaker across everything and one bad dependency takes down calls to healthy ones.

A word on the threshold. A raw count of 5 is a fine start, but under high traffic it trips too eagerly — five failures out of five thousand requests is noise, not an outage. If your volume is high, move to a failure rate over a rolling window (say, open at 50% failures over the last 20 requests) rather than an absolute count. Libraries like opossum give you this out of the box, and for production Node.js I’d reach for it before hand-rolling the breaker above — the code here is to show the mechanics, not to reinvent a solved problem.

What to log, and what it buys you

Resilience you can’t observe is resilience you can’t trust. Every retry and every breaker state change should emit a structured event: which dependency, which attempt number, the status code, and the elapsed time. Feed those into whatever you already run — for us it’s Grafana Loki — and alert on two signals specifically: retry rate climbing, and any breaker sitting open for more than a few seconds.

The retry rate is the early warning. A dependency usually degrades before it dies — you’ll see retries tick up minutes before the breaker trips. That’s your window to page someone while the checkout is still limping along instead of after it’s flat on the floor.

Where the boundary actually goes

One last thing, because it’s the mistake I’d most want to save you from. Retry logic belongs at the client boundary — the code that calls out to a dependency — not scattered through business logic. When retries leak into your domain code, you get retries wrapping retries: your service retries a function that internally retries an HTTP call that sits behind a proxy that also retries. Three layers, each unaware of the others, and a single user click that fans out into dozens of requests hammering an already-sick service.

Put the retry logic, the deadline, the jitter, and the breaker in one place — the API client — and let everything above it call a plain async function that either returns or throws. The resilience lives at the edge. Everything inside stays simple, and stays honest about the fact that the network is the one thing you don’t control.

Ditulis oleh Reza Pradipta