OpenTelemetry di Microservices Jakarta: 18 Bulan

18 bulan lalu, SaaS B2B Jakarta yang saya advise — 22 microservice, ~4.500 RPS peak, tim 24 engineer — pakai Sentry untuk error tracking dan Datadog untuk APM. Datadog bill $2.800/bulan dan growing. Mereka mau eksplor self-host + OpenTelemetry untuk reduce cost dan vendor lock-in.

Hari ini, 18 bulan stable di OTel + self-host Tempo + Loki. Share angka, architecture, dan keputusan operasional yang bikin atau patahkan project ini.

Konteks awal

• Service: 22 microservice (12 Java Spring, 6 Go, 3 Node, 1 Python ML).
• Throughput: 4.500 RPS aggregate peak, 2.1M trace span/menit.
• Infrastructure: Kubernetes EKS Jakarta (ap-southeast-3).
• Existing observability: Datadog APM ($2.800/mo), Sentry ($89/mo), Prometheus self-host.
• Tim: 24 engineer, 3 SRE.

Pain point sebelum migrate:

• Datadog cost overhead growing 18% YoY.
• Vendor lock-in: trace instrumentation pakai DD SDK proprietary.
• Cross-correlation antara DD APM + Sentry + Prometheus manual.

Architecture: OTel Collector + Tempo + Loki + Grafana

[App services with OTel SDK]
        ↓ OTLP/gRPC
[OTel Collector (Agent mode, per-node DaemonSet)]
        ↓
[OTel Collector (Gateway mode, deployment with HPA)]
        ├─ Traces  → Tempo (object storage S3)
        ├─ Metrics → Prometheus (push via remote_write)
        ├─ Logs    → Loki (S3 chunks)
        └─ Sample for billing → Datadog (5% canary, parallel until full migration confidence)
        
[Grafana] ← unified UI for traces/metrics/logs

Per-node OTel Collector (Agent)

DaemonSet untuk reduce network hop dari app pod:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  resourcedetection:
    detectors: [env, system, eks]
  k8sattributes:
    auth_type: serviceAccount

exporters:
  otlp:
    endpoint: otel-gateway.observability.svc:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp]

k8sattributes enrich span dengan metadata pod (namespace, deployment, node) tanpa SDK overhead.

Gateway OTel Collector

Deployment HPA berdasarkan CPU + queue depth:

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 5000
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-policy
        type: latency
        latency: {threshold_ms: 1000}
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: {sampling_percentage: 8}

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.observability.svc:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prom-receiver:9090/api/v1/write
  loki:
    endpoint: http://loki-distributor:3100/loki/api/v1/push

Tail-based sampling = decide after seeing full trace (waited 30s buffer). Vs head-based (decide at root span start), tail catches the trace-error-late-in-flow yang head sampling miss.

Sampling rate effective: 100% error/slow + 8% normal. Total ~12-14% trace persisted. Storage saving signifikan.

Instrumentation per language

Java (Spring Boot)

Javaagent auto-instrumentation:

java -javaagent:/opt/otel/opentelemetry-javaagent.jar \
  -Dotel.service.name=payment-service \
  -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
  -Dotel.resource.attributes=deployment.environment=prod,team=payments \
  -jar app.jar

Auto-instrument: Spring MVC, JDBC, Kafka, Redis, HTTP client. Coverage out-of-box ~85%.

Manual instrumentation untuk business span:

@WithSpan("payment.process")
public PaymentResult process(@SpanAttribute("payment.id") String id, ...) {
    Span span = Span.current();
    span.setAttribute("payment.amount", req.amount());
    span.setAttribute("payment.method", req.method());
    
    var result = paymentGateway.charge(req);
    span.setAttribute("payment.gateway_ref", result.referenceId());
    
    return result;
}

Go

Manual instrumentation (otel-go masih lebih banyak boilerplate dari Java):

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
)

var tracer = otel.Tracer("order-service")

func ProcessOrder(ctx context.Context, req OrderRequest) error {
    ctx, span := tracer.Start(ctx, "order.process",
        trace.WithAttributes(
            attribute.String("order.id", req.ID),
            attribute.Int("order.items", len(req.Items)),
        ))
    defer span.End()
    
    if err := validateInventory(ctx, req); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "inventory check failed")
        return err
    }
    // ...
}

HTTP middleware via otelhttp.NewHandler cover incoming request. SQL via otelsql wrap driver.

Node.js (Bun service)

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4317' }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false }, // too noisy
    }),
  ],
  serviceName: 'notification-service',
});

sdk.start();

Auto-instrument: HTTP, gRPC, Postgres, Redis, Kafka. Bun support OTel via Node SDK compat (since Bun 1.3).

Python (ML pipeline)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317", insecure=True))
)

tracer = trace.get_tracer(__name__)

@tracer.start_as_current_span("ml.inference")
def predict(features):
    span = trace.get_current_span()
    span.set_attribute("model.version", MODEL_VERSION)
    # ...

Backend: Tempo

Tempo (Grafana stack) untuk trace storage. Setup di EKS via Helm:

tempo:
  storage:
    trace:
      backend: s3
      s3:
        bucket: company-tempo-traces
        endpoint: s3.ap-southeast-3.amazonaws.com
        region: ap-southeast-3
  retention: 720h  # 30 days

distributor:
  replicas: 4
  resources:
    requests: { cpu: 1, memory: 4Gi }
    limits: { cpu: 2, memory: 8Gi }

ingester:
  replicas: 6
  resources:
    requests: { cpu: 2, memory: 8Gi }
    limits: { cpu: 4, memory: 16Gi }

compactor:
  replicas: 2

Tempo cost: S3 storage ~$96/bulan untuk 4.2TB (30-day retention setelah sampling). EKS node Tempo: ~$280/bulan (12 vCPU + 48GB total).

Total backend cost: ~$380/bulan.

Trace correlation di Grafana

Grafana 11+ native correlate trace ↔ log ↔ metric. Setup:

datasources:
  - name: Tempo
    type: tempo
    url: http://tempo-query-frontend:3100
    jsonData:
      tracesToLogsV2:
        datasourceUid: 'loki'
        spanStartTimeShift: '-1h'
        spanEndTimeShift: '1h'
        filterByTraceID: true
      tracesToMetrics:
        datasourceUid: 'prometheus'

Workflow: alert fire → klik trace_id → buka di Tempo → klik “View related logs” → muncul di Loki dengan span_id filter. Time-to-context untuk SRE turun signifikan.

Hasil 18 bulan

Metric	Datadog (sebelum)	OTel + Self-host	Delta
Cost monthly	$2.800	$720	-74%
Trace coverage	78% (DD agent)	94%	+16pp
Trace retention	15 hari	30 hari	2x
MTTR (incident)	28 min	12 min	-57%
Vendor lock-in	high	low	-
Cross-correlation effort	manual	unified	-
Storage per month	(vendor managed)	4.2 TB	-

MTTR turun karena: lebih banyak trace context (sampling lebih agresif untuk error), unified UI mengurangi context switch antar tool.

Yang break

1. Cardinality explosion di label

Bulan ke-2, salah satu service set span attribute user_id (UUID). 24k unique user × 8 endpoint = ~190k unique time series. Prometheus remote_write Tempo overload, ingestor OOM.

Fix: convert ke span attribute (per-trace, not metric label). Plus audit otel.metric.attributes di setiap service untuk catch high-cardinality di label.

Rule of thumb yang kami pakai: label = enum-ish (< 100 unique values). Anything user-specific → span attribute, bukan metric label.

2. Tail sampling decision_wait terlalu pendek

decision_wait: 30s initial. Tapi ada trace yang span-nya datang lebih lambat (e.g., async kafka consumer delay). Trace incomplete di sampling decision time → di-sample wrong.

Fix: decision_wait: 90s + num_traces: 200000 (buffer lebih besar). Memory Gateway Collector naik 8GB → 14GB. Acceptable.

3. OTel SDK CPU overhead di hot path

Service Java payment-service: latency p99 naik 8% setelah enable OTel agent. Investigasi: auto-instrument too aggressive di hot path (per-method tracing).

Fix: konfigurasi exclude noisy instrumentation:

-Dotel.instrumentation.commons-dbcp2.enabled=false
-Dotel.instrumentation.servlet.experimental.capture-request-parameters=false

Plus disable JDBC instrumentation untuk metadata query (high volume, low value).

CPU overhead turun ke ~1.2%.

4. Trace context propagation gagal cross Kafka

Kafka producer-consumer di service Java + Go. Context propagation pakai W3C trace context header. Bug di service Go yang pakai library Kafka lama (sarama v1.30) yang tidak inject header.

Fix: upgrade ke kafka-go (resmi support OTel propagator). Migration 2 minggu kerja.

5. Loki query slow saat investigation incident

Loki bisa slow saat query log dengan time range besar (> 6 jam). Penyebab: chunk scan banyak.

Fix:

• Increase chunk_idle_period (chunk lebih besar, fewer chunk).
• Pakai LogQL filter di awal query, bukan di akhir.
• Tier “investigation” Loki dengan SSD lokal, hot data 7 hari.

Query investigation latency p95: 18 detik → 4 detik.

6. Datadog migration overlap cost

Selama 6 bulan migration, kami run dual (Datadog + OTel). Bill DD masih jalan + cost self-host. Periode tumpang tindih total cost: $2.800 + $720 = $3.520/bulan × 6 bulan = $21k.

Worth it untuk confidence parallel run, tapi budget approval butuh framing yang jelas (“invest 6 bulan tumpang tindih, save 24 bulan setelahnya”).

Kapan saya tidak rekomendasi OTel self-host

1. Tim < 10 engineer tanpa SRE dedicated: operational overhead Tempo + Loki + OTel Collector tidak negligible.
2. Workload < 500 RPS aggregate: Datadog Starter cukup, self-host saving tidak besar.
3. Compliance mandate vendor-managed: beberapa audit framework butuh vendor-managed observability.
4. Language Anda OTel SDK belum mature: Erlang, Elixir, Crystal — OTel SDK masih early. Stick dengan APM yang punya native agent.

Verdict

OpenTelemetry + self-host (Tempo + Loki + Grafana) cocok untuk perusahaan Indonesia ukuran mid-market dengan workload 1k+ RPS dan tim engineering yang punya SRE bandwidth. Saving cost signifikan (~75%) plus zero vendor lock-in.

Tapi: setup butuh 6-8 bulan untuk reach stability. Operasional ongoing butuh 0.5 FTE SRE dedicated. Bukan magic.

Untuk konteks single-stack observability lebih murah lihat juga Prometheus Grafana SaaS Jakarta atau LogRocket vs Sentry vs Datadog comparison.