karawaci.kode

2026-09-21 · 9 min

Circuit Breaker Pattern in Node.js with Opossum: From Theory to Production

Your payment gateway goes down at 22:00 on a Friday. Not the whole service — just the settlement endpoint. Every order placed after that still hits it, waits 30 seconds for a timeout, and your Node.js event loop fills up with pending requests. Five minutes later, your checkout service is unresponsive. One downstream dependency took your entire platform with it.

This is the failure mode the circuit breaker pattern exists to prevent. And in 2026, the simplest way to implement it in Node.js is still Opossum — maintained by Red Hat’s NodeShift team, production-proven, and refreshingly un-magical.

What the circuit breaker actually does

The pattern has three states. Closed is normal operation: requests flow through, failures are tracked. Once the error rate crosses a configurable threshold within a rolling window, the circuit opens — requests fail immediately without touching the dependency. After a reset timeout, the circuit enters half-open: one probe request is allowed through. If it succeeds, the circuit closes again. If it fails, the timer resets.

The critical insight is that “fail immediately” is a feature, not a bug. Failing fast lets you return a fallback response, free up your event loop, and prevent cascading failures. It also stops you from adding load to a dependency that’s already struggling.

Installing and wiring up Opossum

npm install opossum
npm install opossum-prometheus prom-client  # for metrics

Basic usage is straightforward:

import CircuitBreaker from "opossum";

// The function you want to protect — must return a Promise
async function callPaymentGateway(payload: PaymentPayload): Promise<PaymentResult> {
  const response = await fetch("https://gateway.example.com/settle", {
    method: "POST",
    body: JSON.stringify(payload),
    headers: { "Content-Type": "application/json" },
  });

  if (!response.ok) {
    throw new Error(`Gateway returned ${response.status}`);
  }

  return response.json();
}

const breaker = new CircuitBreaker(callPaymentGateway, {
  timeout: 5000,               // treat as failure if no response in 5s
  errorThresholdPercentage: 50, // open circuit if >50% of requests fail
  resetTimeout: 30000,          // try again after 30 seconds
  volumeThreshold: 10,          // need at least 10 requests before error % matters
});

// Export the fire function — callers use this, not the raw function
export const settlePayment = (payload: PaymentPayload) => breaker.fire(payload);

One thing that trips people up: you wrap the function at module initialization time, not at call time. The circuit breaker instance is stateful — it tracks errors across calls. If you new CircuitBreaker() inside your request handler, you get a fresh circuit on every request, which defeats the entire purpose.

Fallbacks that are actually useful

An open circuit with no fallback just throws an error. That might be fine for some operations, but for anything user-facing you want a deliberate fallback:

// Fallback receives the same arguments as the original function
breaker.fallback(async (payload: PaymentPayload) => {
  // Queue for async retry — don't block the user
  await paymentQueue.add("retry-settlement", payload, {
    attempts: 3,
    backoff: { type: "exponential", delay: 5000 },
  });

  // Return a pending state so the frontend can poll
  return {
    id: generateId(),
    status: "QUEUED",
    message: "Payment is being processed. You will receive a confirmation shortly.",
  };
});

The fallback runs whenever the circuit is open and whenever the wrapped function throws. This is a subtle but important distinction: your fallback also catches failures during normal (closed) circuit operation unless you listen to events selectively.

For read-heavy operations, the fallback pattern is even simpler — return cached data:

const breaker = new CircuitBreaker(fetchUserProfile, { timeout: 3000 });

breaker.fallback(async (userId: string) => {
  const cached = await redis.get(`user:${userId}`);
  if (cached) return JSON.parse(cached);
  throw new Error("Service unavailable and no cache available");
});

Events and observability

Opossum emits events for every state transition. Wire these into your logger immediately — you want to know the moment a circuit opens in production:

import logger from "./logger"; // your structured logger

breaker.on("open", () => {
  logger.error("circuit_breaker_opened", {
    circuit: "payment-gateway-settlement",
    timestamp: new Date().toISOString(),
  });
});

breaker.on("halfOpen", () => {
  logger.warn("circuit_breaker_half_open", {
    circuit: "payment-gateway-settlement",
  });
});

breaker.on("close", () => {
  logger.info("circuit_breaker_closed", {
    circuit: "payment-gateway-settlement",
  });
});

breaker.on("fallback", (result) => {
  logger.warn("circuit_breaker_fallback_used", {
    circuit: "payment-gateway-settlement",
    fallbackResult: result,
  });
});

For Prometheus metrics, the opossum-prometheus package does the heavy lifting:

import { PrometheusMetrics } from "opossum-prometheus";
import { register } from "prom-client";

// Register the circuit breaker with Prometheus
const metrics = new PrometheusMetrics({ registry: register });
metrics.add([breaker]);

// Mount the /metrics endpoint in your Express app
app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.send(await register.metrics());
});

This gives you four metrics per circuit automatically: fires (total calls), successes, failures, fallbacks, timeouts, and a gauge for whether the circuit is currently open. In Grafana, I set an alert on circuit_breaker_open{circuit="payment-gateway-settlement"} == 1 with a 2-minute evaluation window — long enough to avoid noise from transient half-open states, short enough to catch real outages.

Structuring multiple circuits in a real service

A real backend service usually talks to several dependencies: a payment gateway, a notification service, a third-party KYC provider. Each gets its own circuit breaker with tuned parameters. I keep them in a dedicated module:

// src/infrastructure/circuit-breakers.ts
import CircuitBreaker from "opossum";
import { PrometheusMetrics } from "opossum-prometheus";
import { register } from "prom-client";

const metrics = new PrometheusMetrics({ registry: register });

function makeBreaker<TArgs extends unknown[], TResult>(
  name: string,
  fn: (...args: TArgs) => Promise<TResult>,
  options?: CircuitBreaker.Options
): CircuitBreaker<TArgs, TResult> {
  const breaker = new CircuitBreaker(fn, {
    name,
    timeout: 5000,
    errorThresholdPercentage: 50,
    resetTimeout: 30000,
    volumeThreshold: 10,
    ...options,
  });

  // Attach logging for every circuit
  breaker.on("open", () => logger.error(`circuit_open: ${name}`));
  breaker.on("close", () => logger.info(`circuit_closed: ${name}`));

  metrics.add([breaker]);
  return breaker;
}

export const paymentGatewayBreaker = makeBreaker(
  "payment-gateway",
  callPaymentGateway,
  { timeout: 8000 } // payment gateway is known to be slow
);

export const notificationBreaker = makeBreaker(
  "notification-service",
  sendNotification,
  {
    timeout: 2000,       // notifications should be fast
    resetTimeout: 15000, // recover faster since it's not critical path
  }
);

export const kycBreaker = makeBreaker("kyc-provider", callKycProvider, {
  errorThresholdPercentage: 30, // KYC failures are expensive — open circuit sooner
  resetTimeout: 60000,          // KYC provider recovery is typically slow
});

The factory function makeBreaker eliminates repetition while letting each circuit have its own tuned parameters. This matters in production: a notification service should recover faster and tolerate more failures than a payment gateway.

Health check endpoint

Expose circuit state in your health check so your load balancer and orchestration layer can see it:

app.get("/health", (req, res) => {
  const circuits = {
    paymentGateway: {
      open: paymentGatewayBreaker.opened,
      halfOpen: paymentGatewayBreaker.halfOpen,
      stats: paymentGatewayBreaker.stats,
    },
    notification: {
      open: notificationBreaker.opened,
      halfOpen: notificationBreaker.halfOpen,
      stats: notificationBreaker.stats,
    },
  };

  const anyOpen = Object.values(circuits).some((c) => c.open);

  res.status(anyOpen ? 503 : 200).json({
    status: anyOpen ? "degraded" : "ok",
    circuits,
  });
});

Whether to return 503 when a circuit opens is a judgment call. For non-critical dependencies, I return 200 with status: "degraded". For payment processing — which handles transactions for merchants across Tangerang, from the shophouses along Jalan MH Thamrin BSD to the logistics companies in the Sinar Mas Land kawasan — a 503 is appropriate when the settlement circuit opens, because we cannot accept orders we cannot eventually process.

What to actually tune

Default values in Opossum are reasonable starting points but almost never correct for your specific dependency. Three parameters matter most:

timeout — set this to your dependency’s P99 latency plus a small buffer, not an arbitrary round number. If your payment gateway’s P99 is 1.8 seconds under normal load, set timeout: 2500. Setting it to 30 seconds “to be safe” means your event loop queues up for 30 seconds before the circuit decides something is wrong.

volumeThreshold — the minimum number of requests in the rolling window before the error percentage is evaluated. Set this too low and a single failure at low traffic opens your circuit. Set it too high and the circuit never opens during a partial outage that only affects a fraction of requests. I start at 10 for services with steady traffic, lower for low-volume dependencies.

resetTimeout — how long the circuit stays open before attempting a probe. The instinct is to set this low so recovery is fast. The problem is that recovery is rarely instant: if your dependency is restarting, 5 seconds isn’t enough for it to finish startup. 30 seconds is a safer default; 60 seconds for dependencies with known slow restarts.

The failure mode Opossum doesn’t cover

Circuit breakers protect against downstream failures. They do not protect against resource exhaustion in your own service. If your circuit is closed and your dependency starts responding in 4.9 seconds (just under your 5-second timeout), you’ll accumulate a large number of in-flight requests waiting for responses. For this, you need a concurrency limit on the circuit:

const breaker = new CircuitBreaker(callPaymentGateway, {
  timeout: 5000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
  volumeThreshold: 10,
  capacity: 50, // reject new requests when 50 are already in-flight
});

The capacity option in Opossum puts a hard cap on concurrent in-flight requests. Requests that arrive when the circuit is at capacity are rejected immediately — which is exactly the behavior you want. A 429-equivalent fast rejection is far better than accumulating 500 waiting connections.

Circuit breakers are not a silver bullet, and Opossum won’t automatically make your service resilient. You still need to think carefully about which fallback behavior makes sense for each dependency, tune your thresholds based on real traffic patterns, and — critically — test what actually happens when you force a circuit open in staging. Most teams skip that last step and discover their fallback logic has a bug at 22:00 on a Friday.

Ditulis oleh Reza Pradipta