karawaci.kode

2026-09-08 · 11 min

Distributed Tracing in Node.js: Jaeger vs Tempo, OpenTelemetry Setup, and What Actually Works in 2026

Your Node.js service throws a 3-second timeout every few minutes in production, and console.log hasn’t told you anything useful in three days. Distributed tracing is the answer — and getting it wired up correctly from the start saves you from the two-week debugging session that comes from doing it halfway.

This article covers the full setup: instrumenting a Node.js service with OpenTelemetry, choosing between Jaeger and Grafana Tempo as a backend, configuring the Collector, and the sampling strategy that doesn’t bankrupt your storage budget.

Why distributed tracing, and why OpenTelemetry

Before OpenTelemetry, instrumentation was vendor-specific. Switching from Jaeger to Zipkin meant ripping out the Jaeger client and rewriting every instrumentation call. OpenTelemetry solves this by being the instrumentation layer — your code emits OTLP (OpenTelemetry Protocol) spans, and you route those wherever you want at the Collector level.

In practice: you instrument once, and the decision of “Jaeger or Tempo or Honeycomb” becomes a config change in the Collector, not a code change in forty microservices.

The OpenTelemetry Node.js SDK is production-stable in 2026. The @opentelemetry/sdk-node package covers auto-instrumentation for Express, Fastify, gRPC, HTTP, database clients, and more. Write zero manual span code for most common operations.

Instrumenting a Node.js service

Install the core packages:

npm install \
  @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-grpc \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions

Create a dedicated tracing bootstrap file — do not put this inside your main application file:

// src/tracing.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { Resource } from "@opentelemetry/resources";
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION } from "@opentelemetry/semantic-conventions";
import { ParentBasedSampler, TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";

const exporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4317",
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: process.env.SERVICE_NAME ?? "payment-service",
    [SEMRESATTRS_SERVICE_VERSION]: process.env.APP_VERSION ?? "unknown",
    "deployment.environment": process.env.NODE_ENV ?? "development",
  }),
  traceExporter: exporter,
  instrumentations: [
    getNodeAutoInstrumentations({
      // Disable filesystem instrumentation — too noisy, not useful
      "@opentelemetry/instrumentation-fs": { enabled: false },
      // pg instrumentation: capture query but not the full parameters by default
      "@opentelemetry/instrumentation-pg": {
        enhancedDatabaseReporting: false,
      },
    }),
  ],
  sampler: new ParentBasedSampler({
    // Sample 20% of traces in production; parent decision is respected
    root: new TraceIdRatioBasedSampler(
      process.env.TRACE_SAMPLE_RATIO ? parseFloat(process.env.TRACE_SAMPLE_RATIO) : 0.2
    ),
  }),
});

sdk.start();

process.on("SIGTERM", () => {
  sdk.shutdown().finally(() => process.exit(0));
});

Then in your entrypoint, require this file before anything else:

// src/index.ts
import "./tracing"; // must be first import
import express from "express";
// ... rest of your app

Or via the --require flag if you prefer not touching the entrypoint:

node --require ./dist/tracing.js dist/index.js

The --require approach is cleaner for brownfield services where you don’t want to touch the existing entrypoint.

Adding manual spans for business context

Auto-instrumentation captures HTTP requests, database queries, and external calls. It does not know what “payment processed” means in your domain. Add manual spans for operations that matter:

import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("payment-service");

async function processPayment(orderId: string, amount: number): Promise<PaymentResult> {
  return tracer.startActiveSpan("payment.process", async (span) => {
    span.setAttributes({
      "payment.order_id": orderId,
      "payment.amount_idr": amount,
      "payment.provider": "midtrans",
    });

    try {
      const result = await midtransClient.charge({ orderId, amount });
      span.setAttributes({ "payment.transaction_id": result.transactionId });
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Two things worth noting here: always call span.end() in a finally block, and always call span.recordException() before span.setStatus(ERROR) — the exception gives you the stack trace, the status gives Jaeger/Tempo a signal to flag the trace.

Jaeger vs Grafana Tempo: which to pick

Both receive OTLP/gRPC. The choice is about your existing stack, not tracing capability.

Pick Jaeger if:

  • You want an all-in-one tracing stack without committing to Grafana
  • Your team is already familiar with the Jaeger UI
  • You’re running on Kubernetes and want the Jaeger Operator to manage the lifecycle

Pick Grafana Tempo if:

  • You’re already running Grafana, Loki, and Prometheus (the LGTM stack)
  • You want traces correlated with logs (Loki) and metrics (Mimir) in a single Grafana dashboard
  • Storage cost matters — Tempo uses object storage (MinIO, S3, GCS), which is dramatically cheaper than Elasticsearch for Jaeger at scale

At the office in Serpong — serving traffic from users across Tangerang Selatan — we switched from Jaeger + Elasticsearch to Tempo + MinIO six months ago. Storage cost dropped by roughly 70% for the same trace retention period. The Grafana correlation between a slow trace and the corresponding Loki log line is genuinely useful for on-call debugging.

OpenTelemetry Collector configuration

Do not export directly to Jaeger or Tempo in production. Run an OTel Collector as a sidecar or DaemonSet:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

  # Enrich spans with Kubernetes metadata
  k8sattributes:
    auth_type: serviceAccount
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.deployment.name

  # Tail-based sampling: keep all error traces, sample 10% of healthy ones
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 500
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow-traces
        type: latency
        latency: { threshold_ms: 2000 }
      - name: sample-healthy
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  # For Tempo
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  # For Jaeger (uncomment if using Jaeger instead)
  # jaeger:
  #   endpoint: jaeger:14250
  #   tls:
  #     insecure: true

  # Debug: log to stdout in dev
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, batch, tail_sampling]
      exporters: [otlp/tempo]

The tail_sampling policy here keeps 100% of traces with errors or latency above 2 seconds, and samples 10% of everything else. Adjust expected_new_traces_per_sec and num_traces to match your actual throughput — undersizing these causes the Collector to drop traces before the sampling decision is made.

Connecting traces to logs with correlation IDs

The trace is only half the picture if you can’t cross-reference it with logs. Add the trace ID and span ID to every log line:

import { trace } from "@opentelemetry/api";
import pino from "pino";

const logger = pino();

function getTraceContext() {
  const span = trace.getActiveSpan();
  if (!span) return {};
  const ctx = span.spanContext();
  return {
    traceId: ctx.traceId,
    spanId: ctx.spanId,
  };
}

// Wrap pino to inject trace context automatically
export function createLogger(name: string) {
  return {
    info: (msg: string, data?: object) =>
      logger.info({ ...getTraceContext(), ...data, service: name }, msg),
    error: (msg: string, data?: object) =>
      logger.error({ ...getTraceContext(), ...data, service: name }, msg),
    warn: (msg: string, data?: object) =>
      logger.warn({ ...getTraceContext(), ...data, service: name }, msg),
  };
}

In Grafana, you can configure a “Derived Field” on the Loki datasource to turn traceId fields in log lines into clickable links that open the trace in Tempo. This is the cross-signal correlation that makes the LGTM stack genuinely useful on-call — you see the error in Loki, click the trace ID, and land on the full distributed trace in Tempo showing exactly which downstream call was slow.

Sampling strategy that doesn’t wreck your storage

A common mistake: setting TraceIdRatioBasedSampler(1.0) in development because “I want to see all traces,” then forgetting to change it before deploying to production. At 500 req/s, 100% head-based sampling generates enough trace data to fill your Elasticsearch or Tempo storage in days.

The right approach for 2026:

EnvironmentHead samplingTail sampling (Collector)
Local dev100%None
Staging100%None
Production20%Keep errors + slow traces

Head sampling at 20% means the application only sends 1 in 5 traces to the Collector. The Collector’s tail sampling then ensures that within those 20%, all error and slow traces are kept. This two-stage approach keeps storage costs bounded while guaranteeing you never lose a trace that would have been useful for debugging.

One thing that catches people: ParentBasedSampler respects the sampling decision of the upstream service. If your load balancer or API gateway starts a trace and samples it out, all downstream services will also drop it — even if your Node.js service has its own 100% sampler configured. Make sure your propagation and sampling decisions are consistent across the entire call chain.

What to check when traces look incomplete

Missing spans almost always come down to three things:

1. An uninstrumented service in the chain. If service A calls service B and B is not instrumented, the trace ends at the A→B boundary. The span for the outgoing HTTP call from A still appears, but there are no spans for what happens inside B.

2. Propagation format mismatch. OpenTelemetry defaults to W3C Trace Context (traceparent header). If your service calls a legacy service that reads B3 headers (X-B3-TraceId, X-B3-SpanId), the trace is broken at that boundary. Fix it by registering both propagators:

import { CompositePropagator, W3CTraceContextPropagator } from "@opentelemetry/core";
import { B3Propagator, B3InjectEncoding } from "@opentelemetry/propagator-b3";

// In SDK initialization:
textMapPropagator: new CompositePropagator({
  propagators: [
    new W3CTraceContextPropagator(),
    new B3Propagator({ injectEncoding: B3InjectEncoding.MULTI_HEADER }),
  ],
}),

3. The Collector dropping spans under load. Check the Collector’s own metrics endpoint (/metrics on port 8888 by default). Look at otelcol_processor_dropped_spans_total — if this is climbing, increase num_traces in the tail sampler or scale the Collector horizontally.

Production checklist

Before shipping this to production, verify:

  • SERVICE_NAME is set correctly per service — traces from three services with the same service name are impossible to filter in Jaeger/Tempo
  • APP_VERSION is injected at deploy time (from the image tag or git SHA), not hardcoded
  • The Collector has resource limits set — it will OOM under sustained high cardinality if num_traces is too large for the available memory
  • OTLP endpoint is using gRPC (port 4317), not HTTP (4318), for the exporter — gRPC is more efficient for high-volume trace export
  • Sensitive data is not being captured in span attributes — database queries with user PII, payment card numbers in HTTP body spans

Distributed tracing pays off when the setup is complete enough that you actually trust the data. A half-instrumented system where some services have traces and others don’t trains your team to not use the tool, which defeats the investment entirely. Instrument everything, or at minimum every service in your critical path, before declaring it production-ready.

Ditulis oleh Reza Pradipta