karawaci.kode

2026-07-14 · 8 min

Distributed Tracing OTEL di Microservices BCA-like: Lessons

Setahun lalu saya jadi consultant arsitektur untuk implementasi distributed tracing di bank besar Indonesia (tidak BCA exact, tapi kelasnya mirip — top 10 bank umum, 84 microservice di domain core banking + digital channel, ~28k span/detik aggregate). Saya tidak akan share nama, tapi share angka, pattern, dan lessons yang transferable.

Konteks (digeneralisasi untuk privacy)

  • Industri: bank umum top-10 Indonesia.
  • Skala: 84 microservice, tim 240 engineer (15 squad), 4 datacenter Jabodetabek.
  • Workload: core banking (account, transaction, loan, deposit) + digital channel (mobile banking, internet banking, API for partners).
  • Compliance: BI (Bank Indonesia) regulation, ISO 27001, PCI-DSS Level 1, SOC 2 Type 2.
  • Throughput: peak 4.2k TPS aggregate di payment flow, 28k span/detik trace volume.
  • Existing observability: AppDynamics (kontrak akan habis), Splunk untuk log, custom dashboard untuk metric.

Pain point sebelum project:

  1. Vendor lock-in AppDynamics: instrumentation specific, sulit migrate.
  2. Trace incomplete cross-service: payment flow hit 12+ service, trace pecah di service boundary lama (mostly Java legacy + beberapa Go newer).
  3. MTTR tinggi: incident “transaksi gagal di tahap X” butuh manual correlation 30-90 menit.
  4. PII di trace: ada NIK, account number, NPWP keluar di trace attribute (audit flag).

Strategi: OTel sebagai standard, hybrid backend

Saya tidak rekomendasi langsung kill AppDynamics. Strategi:

  1. OTel SDK di semua service (standardize instrumentation).
  2. Dual-export ke AppDynamics (legacy backend, masih ada license) + Tempo self-host (target backend).
  3. Validate parity 6 bulan, lalu sunset AppDynamics.
  4. Compliance review PII scrubbing sebelum production.

Total estimated timeline: 18 bulan.

Architecture OTel di bank ini

[84 microservices]
    ↓ OTLP/gRPC
[OTel Collector Agent (DaemonSet per node, 240 node)]
    ↓
[OTel Collector Gateway (HPA, 12 instance avg)]
    ├─ PII scrubbing (regex + ML-based)
    ├─ Tail sampling (12% hit ratio)
    ├─ Export to Tempo (S3 backend)
    └─ Dual-export to AppDynamics (during migration)
    
[Tempo distributor → ingester → compactor]
    ↓
[S3-compatible (MinIO on-prem)]
    ↓ query
[Grafana Tempo UI]

Why on-prem MinIO bukan AWS S3?

Bank Indonesia regulation: data residency. Trace data berisi info transaksi (meskipun PII di-scrub, masih sensitive). Harus on-prem DC Indonesia.

MinIO cluster: 6 node, 4 disk × 16TB = 384TB raw, ~256TB usable dengan erasure coding EC:4+2.

PII scrubbing critical

Multi-layer:

Layer 1: SDK level (per-service)

@Configuration
public class OtelConfig {
    
    @Bean
    public OpenTelemetry openTelemetry() {
        SpanProcessor scrubProcessor = new PiiScrubSpanProcessor();
        
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(scrubProcessor)
            .addSpanProcessor(BatchSpanProcessor.builder(otlpExporter).build())
            .build();
        
        return OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
            .build();
    }
}

public class PiiScrubSpanProcessor implements SpanProcessor {
    
    // NIK: 16 digit, NPWP: 15-16 digit dengan format, account: 10-13 digit
    private static final Pattern NIK = Pattern.compile("\\b\\d{16}\\b");
    private static final Pattern NPWP = Pattern.compile("\\b\\d{2}\\.\\d{3}\\.\\d{3}\\.\\d-\\d{3}\\.\\d{3}\\b");
    private static final Pattern ACCOUNT = Pattern.compile("\\b\\d{10,13}\\b");
    
    @Override
    public void onEnd(ReadableSpan span) {
        // Mutate before export
        span.getAttributes().forEach((key, value) -> {
            if (value instanceof String s) {
                String scrubbed = scrub(s);
                if (!scrubbed.equals(s)) {
                    ((ReadWriteSpan) span).setAttribute(key, scrubbed);
                }
            }
        });
    }
    
    private String scrub(String input) {
        input = NIK.matcher(input).replaceAll("[NIK_REDACTED]");
        input = NPWP.matcher(input).replaceAll("[NPWP_REDACTED]");
        input = ACCOUNT.matcher(input).replaceAll(m -> {
            String s = m.group();
            return s.substring(0, 4) + "***" + s.substring(s.length() - 3);
        });
        return input;
    }
}

Layer 2: OTel Collector (defense in depth)

processors:
  redaction:
    allow_all_keys: true
    blocked_values:
      - "[1-9]\\d{15}"  # NIK pattern
      - "\\d{2}\\.\\d{3}\\.\\d{3}\\.\\d-\\d{3}\\.\\d{3}"  # NPWP
    summary: silent

Layer 3: audit script harian

Cron daily yang query Tempo dengan regex match PII pattern. Kalau ditemukan: incident report ke compliance team.

#!/bin/bash
# audit-pii-traces.sh
# Run daily 03:00
COUNT=$(tempo-cli query --search 'attribute=~"\\b\\d{16}\\b"' --since=24h | wc -l)
if [ "$COUNT" -gt 0 ]; then
    curl -X POST $COMPLIANCE_WEBHOOK -d "{\"alert\":\"PII detected in traces\",\"count\":$COUNT}"
fi

Tail-based sampling tuning

Default 100% sampling = ~28k span/detik × storage cost prohibitive. Tail sampling:

processors:
  tail_sampling:
    decision_wait: 60s
    num_traces: 200000
    expected_new_traces_per_sec: 5000
    policies:
      - name: error-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      
      - name: slow-policy
        type: latency
        latency: {threshold_ms: 800}
      
      - name: high-value-flow
        type: string_attribute
        string_attribute:
          key: business.flow
          values: ["payment", "transfer", "loan-disbursement"]
      
      - name: rare-endpoint
        type: probabilistic
        probabilistic: {sampling_percentage: 25}
        and_sub_policy:
          name: rare-tag
          type: string_attribute
          string_attribute: {key: endpoint.popularity, values: ["rare"]}
      
      - name: baseline-probabilistic
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Effective sampling rate:

  • Error trace: 100%
  • Slow trace (>800ms): 100%
  • High-value business flow (payment, transfer, loan): 100%
  • Rare endpoint: 25%
  • Everything else: 5%

Aggregate observed: 12% trace persisted to storage. ~3.36k span/detik sustained = ~290M span/day = ~110GB compressed/day = ~3.3TB/month.

30-day retention: ~10TB storage. Fit di MinIO cluster comfortably.

Cross-service context propagation

W3C Trace Context standard. Tapi banyak service Java legacy pakai traceparent header lama atau custom format.

Solusi: OTel Collector punya Receivers yang bisa parse multiple format dan normalize ke W3C.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  zipkin:
    endpoint: 0.0.0.0:9411  # untuk service yang masih Zipkin format
  jaeger:
    protocols:
      thrift_http:
        endpoint: 0.0.0.0:14268  # legacy Jaeger

Plus per-service: SDK pakai composite propagator yang try multiple:

.setPropagators(ContextPropagators.create(
    TextMapPropagator.composite(
        W3CTraceContextPropagator.getInstance(),
        B3Propagator.injectingMultiHeaders()
    )
))

Hasil 12 bulan

MetricSebelum (AppDynamics)Sesudah (OTel + Tempo)
Trace coverage (cross-service complete)62%96%
MTTR (incident detection-to-resolution)42 menit avg12 menit avg
Cost per month$14k (AppDynamics)$1.2k (Tempo + MinIO)
PII leak incident3 (in last 12 mo before)0
Vendor lock-inhighlow
Tim onboard time (new service to traced)2 minggu1 hari

MTTR turun 71% karena: lebih banyak trace context (sampling lebih agresif untuk error), Grafana correlate trace ↔ log ↔ metric unified UI mengurangi context switch.

Use case real: payment flow incident

Insiden bulan ke-7: customer report transfer “Lambat” untuk amount > Rp 100 juta selama 2 jam.

Workflow investigation:

  1. Alert fire: payment.latency.p99 > 5s for amount > 100M.
  2. Click trace_id sample yang trigger alert → buka di Tempo.
  3. Trace span hierarchy:
    • payment-bff (320ms) →
    • account-service (180ms) →
    • ledger-service (140ms) →
    • aml-screening-service (4.2 detik ← bottleneck!)
  4. Drill ke aml-screening-service span detail:
    • Child span external-call.dukcapil-api: 4.1 detik.
  5. Issue: integrasi Dukcapil API untuk enhanced screening high-amount lambat. Vendor partner kena DDoS pagi tadi.
  6. Mitigation: temporary fallback ke screening internal saja untuk amount < Rp 500 juta.

Total time-to-root-cause: 14 menit. Sebelum OTel: estimasi 90 menit (manual log correlation).

Yang break

1. AppDynamics dual-export overhead

Initial setup dual-export ke AppDynamics + Tempo bikin SDK overhead 2x. CPU service-service naik ~3-5%.

Fix: dual-export di OTel Collector level, bukan SDK level. SDK push ke Collector once, Collector fan-out ke 2 destination.

2. Sampling decision lag spike

decision_wait: 60s window. Saat traffic spike, Collector buffer overflow, drop trace.

Fix:

  • Scale up Collector HPA berdasarkan queue depth metric (bukan cuma CPU).
  • Increase num_traces: 500000 (lebih besar in-memory buffer, lebih banyak memory).

Collector resource per pod jadi 4 CPU + 16GB. 12 pod = 48 CPU, 192GB. Acceptable.

3. JVM agent + Spring Boot 2.x compatibility

Beberapa service legacy Spring Boot 2.7 dengan Java 11. OTel agent latest support Java 11 tapi ada quirk di reflection di Spring Boot 2.7 (security manager).

Fix: upgrade service-service Java 17 dulu sebelum enable OTel, atau pakai OTel agent versi LTS (1.32.x) yang compatible dengan Java 11.

4. Tempo compactor lag

Bulan ke-9, compactor mulai lag. Trace search untuk time range > 2 hari slow karena banyak block yang belum compacted.

Fix:

  • Scale compactor dari 2 ke 4 replica.
  • Tune compaction.compacted_block_retention dan compaction.block_retention.

5. Cardinality lagi (selalu cardinality)

Salah satu service set span attribute request.body_hash. Unique per request. Cardinality di Tempo index explode, query slow.

Fix: review semua high-cardinality attribute, mark sebagai “trace_only” (tidak di-index). Tempo support trace_qualifier_attribute config untuk separate indexed vs unindexed attribute.

Pelajaran untuk implementasi bank-scale

1. Compliance team adalah engineering team paling penting di project ini

PII scrubbing bukan afterthought. Saya spend 6 minggu diskusi dengan compliance team untuk define what’s PII, what’s quasi-identifier (cuma sebagian PII tapi sensitive saat combine), how to audit.

2. Dual-stack period tidak negotiable

Tidak ada “big bang switch” dari AppDynamics ke OTel. 6 bulan minimum dual-stack untuk parity validation. Cost extra periode itu adalah insurance.

3. SDK overhead matter di hot path

Service core banking (account, ledger) sensitif terhadap latency. SDK overhead 1ms per span = significant di flow dengan 12 hop. Saya tune sampling decision lebih agresif di SDK level untuk service ini (5% baseline, 100% error).

4. Operational ownership clarity

Trace data baru = pertanyaan baru. Siapa investigate kalau trace incomplete? Siapa tune sampling? Siapa audit PII? Saya define RACI matrix sebelum production. Tanpa itu, trace data jadi “punya everyone, dimiliki no one”.

Verdict

OpenTelemetry di bank Indonesia: feasible, dengan disiplin PII scrubbing + compliance integration dari awal. Self-host backend cost 10-12x lebih murah dari vendor APM komersial dengan compliance posture lebih baik (data residency on-prem).

Tapi: butuh tim observability dedicated (3-4 FTE), bukan side-project. Untuk bank kecil (BPR) atau institusi non-tier-1: vendor managed mungkin lebih cocok.

Bukan magic. Implementasi 18 bulan, 240 engineer affected, 4 compliance review formal. Worth, tapi mata harus terbuka.

Lihat juga OpenTelemetry di SaaS Jakarta untuk konteks SMB-scale OTel deployment.

Ditulis oleh Reza Pradipta