karawaci.kode

2026-07-09 · 8 min

Kafka vs NATS Jetstream: Enterprise Pilih Mana 2026

Setahun lalu, klien enterprise Indonesia (retail omnichannel, 1.200 store + e-commerce, ~12 juta transaksi/bulan) bingung pilih message broker untuk event-driven architecture baru. Ada legacy RabbitMQ yang struggling, dan tim mau modernize. Saya jalankan benchmark Kafka 3.8 (via MSK) vs NATS Jetstream 2.11 self-host, dan akhirnya rekomendasi kombinasi: Kafka untuk core event log, NATS untuk command + low-latency request-reply.

Share angka, trade-off, dan kapan satu pilihan menang telak.

Konteks evaluation

  • Use case:
    • Order event log (~85k order/jam peak, 240k msg/s aggregate dengan downstream fanout)
    • Inventory update (low-latency, < 50ms p99 desired)
    • Notification dispatch (best-effort, retry tolerance)
    • Audit log (mandatory durability, 90-day retention)
  • Skala: 1.200 store + 6 region datacenter (Jabodetabek + 3 luar Jakarta).
  • Compliance: PCI-DSS Level 2 (e-commerce side), regular BI audit (financial reporting).
  • Tim: 14 backend engineer + 3 SRE.
  • Budget broker: < $3.500/bulan production.

Setup benchmark

Hardware identical:

  • 3 broker node m6i.2xlarge (8 vCPU, 32GB RAM, 1TB gp3 EBS @ 3000 IOPS).
  • 3 producer node c6i.large.
  • 3 consumer node c6i.large.
  • Network: same VPC, 25 Gbps baseline.

Test workload:

  • Message size: 1 KB (typical event payload).
  • Producer concurrency: 100 thread total.
  • Consumer concurrency: 100 thread total.
  • Test duration: 4 jam sustained per test scenario.

Software:

  • Kafka 3.8.0 (KRaft mode, no Zookeeper).
  • NATS Jetstream 2.11.2 (clustered, R3 replication).

Throughput

Test 1: max throughput, ack=1 (Kafka) / no-wait (NATS):

MetricKafka 3.8NATS Jetstream 2.11
Throughput sustained240k msg/s180k msg/s
Producer latency p504ms2ms
Producer latency p9918ms8ms
End-to-end latency p5012ms6ms
End-to-end latency p9948ms22ms
Broker CPU avg68%42%
Broker memory18GB11GB
Disk write rate280 MB/s220 MB/s

NATS menang di latency, Kafka menang di sustained throughput dengan margin signifikan.

Test 2: durability mode (Kafka ack=all + min.insync.replicas=2, NATS R3 + ack):

MetricKafka 3.8NATS Jetstream 2.11
Throughput95k msg/s78k msg/s
Producer latency p9938ms18ms
End-to-end latency p9984ms42ms

Kafka penalty durability lebih besar karena 3-replica replication strict semantics. NATS Jetstream R3 quorum-based, lebih cepat tapi semantics sedikit beda (akan saya bahas di bawah).

Operational complexity

Kafka

  • Setup: MSK managed atau self-host.
    • MSK: $1.4k/bulan minimum (3 broker m5.large), scale dengan ukuran.
    • Self-host: butuh KRaft mode (or Zookeeper legacy), backup automation, broker rebalance saat scale.
  • Tooling ekosistem: kafka-ui, Schema Registry, Connect, Streams. Sangat matang.
  • Monitoring: JMX export ke Prometheus, dashboard Grafana standard ada banyak.
  • Operasional overhead: rebalance saat broker add/remove (manual coordination), log retention tuning per topic, partition count planning.
  • Compaction: native, untuk topic dengan key-based deduplication.

NATS Jetstream

  • Setup: 3 node binary, no external dependency.
    • Self-host: ~$280/bulan untuk 3 t3a.medium (instance kecil cukup, NATS efisien).
  • Tooling ekosistem: NATS CLI, NATS box, nats-surveyor. Lebih kecil dari Kafka ekosistem.
  • Monitoring: built-in /varz, /jsz HTTP endpoint, Prometheus exporter.
  • Operasional overhead: jauh lebih kecil. Cluster heal sendiri saat node restart, no rebalance ceremony.
  • Compaction: ada (MaxMsgsPerSubject, MaxBytes), tapi semantik berbeda dari Kafka.

Cost monthly comparison (production-ready):

  • MSK 3-broker production-grade: $2.800/bulan (m5.large + storage + data transfer).
  • NATS Jetstream 3-node self-host: $280/bulan + 0.2 FTE SRE.

Semantics & guarantee

Kafka

  • Ordering: per-partition strict ordering. Cross-partition no ordering guarantee.
  • Delivery: at-least-once default. Exactly-once via transactional producer + idempotent consumer (kompleks tapi ada).
  • Retention: time-based + size-based per topic. Compaction by key.
  • Replay: consumer can seek to arbitrary offset. Time travel debug strong.

NATS Jetstream

  • Ordering: per-stream strict ordering. Cross-stream no ordering.
  • Delivery: at-least-once default. Exactly-once via msg ID dedup window (24 jam default).
  • Retention: time-based, size-based, atau interest-based.
  • Replay: bisa, tapi dengan consumer durable yang persist position.

Untuk audit log dengan compliance:

  • Kafka: log compaction by key, retention indefinite kalau perlu. Format format on-disk well-documented for forensic.
  • NATS Jetstream: bisa, tapi tooling forensic kurang matang.

Untuk fintech audit BI yang minta “tunjukkan transaksi X di posisi Y di event log”: Kafka lebih nyaman.

Yang akhirnya kami pilih: hybrid

Setelah 6 minggu evaluation + PoC, rekomendasi saya:

Use casePilihanAlasan
Order event logKafkaDurability + ecosystem (Kafka Connect ke Postgres, Snowflake)
Audit logKafkaCompaction by key, retention long-term, audit tooling
Inventory updateNATS JetstreamLow latency desired, ack semantics simple
Notification dispatchNATS JetstreamBest-effort, retry pattern simple
Cache invalidationNATS Core (no Jetstream)Pub/sub real-time, no durability needed
Service-to-service commandNATS request-replyRPC-like, low latency

Hybrid berarti ops harus handle 2 system. Trade-off accepted karena per-system simpler dari handle “satu broker untuk semua” yang akhirnya kompleks.

Implementation detail

Kafka cluster

MSK Cluster di ap-southeast-3:

  • 3 broker m5.large
  • KRaft mode (Kafka 3.8 production-ready KRaft)
  • Encryption in-transit (mTLS) + at-rest (KMS)
  • IAM authentication untuk consumer groups
  • Auto-scaling storage enabled

Producer config (Java Spring Boot service):

spring:
  kafka:
    producer:
      bootstrap-servers: ${KAFKA_BROKERS}
      acks: all
      retries: 5
      properties:
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5
        compression.type: zstd
        linger.ms: 10
        batch.size: 65536

Consumer config:

spring:
  kafka:
    consumer:
      bootstrap-servers: ${KAFKA_BROKERS}
      group-id: order-processor-v2
      enable-auto-commit: false
      isolation-level: read_committed
      max-poll-records: 500
      properties:
        partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor

NATS Jetstream cluster

3 node Jetstream R3, di EC2 t3a.medium:

# nats-server.conf
server_name: nats-1
listen: 0.0.0.0:4222
http_port: 8222

cluster {
  name: nats-prod
  listen: 0.0.0.0:6222
  routes: [
    nats-route://nats-2:6222
    nats-route://nats-3:6222
  ]
}

jetstream {
  store_dir: /var/lib/nats/jetstream
  max_memory_store: 4G
  max_file_store: 200G
}

Stream definition (inventory):

nats stream add inventory-updates \
  --subjects "inv.>" \
  --storage file \
  --replicas 3 \
  --retention limits \
  --max-age 24h \
  --max-bytes 50G \
  --dupe-window 5m

Consumer (Go service):

js, _ := nc.JetStream()
sub, _ := js.PullSubscribe("inv.update", "inv-processor",
    nats.AckExplicit(),
    nats.ManualAck(),
    nats.MaxAckPending(1000),
)

for {
    msgs, _ := sub.Fetch(100, nats.MaxWait(5*time.Second))
    for _, msg := range msgs {
        if err := process(msg); err != nil {
            msg.Nak() // retry
            continue
        }
        msg.Ack()
    }
}

Hasil 12 bulan post-migration

MetricRabbitMQ (lama)Kafka + NATS (baru)
Throughput peak28k msg/s240k msg/s
Latency p99 (notification)320ms22ms
Cost monthly$1.200$3.080 (Kafka MSK) + $280 (NATS)
Operational incident4-6/bulan1.5/bulan avg
Data loss incident2 (cluster split brain)0
Tim SRE bandwidth1.5 FTE0.8 FTE

Cost naik (dari RabbitMQ basic ke MSK production-grade). Tapi ops bandwidth saving + throughput headroom worth.

Yang break

1. Kafka partition imbalance

Initial setup: 12 partition per topic, hash by order_id. Beberapa store di kota besar (Jakarta, Surabaya) dominate traffic — partition assignment uneven, beberapa broker CPU 85% sementara yang lain 30%.

Fix: re-key dengan composite key ({region}-{order_id}), re-partition ke 24 partition. CPU spread dari ±55% standard deviation ke ±12%.

2. NATS Jetstream consumer slow recovery

Saat consumer restart, NATS replay dari last ack position. Untuk stream besar dengan banyak unacked, replay bisa lambat (10-30 detik).

Fix: MaxAckPending(1000) (sebelumnya 10000) plus checkpoint position lebih sering (AckWait(5*time.Second) agar redelivery cepat).

3. MSK cross-AZ data transfer cost

MSK 3-broker tersebar multi-AZ. Inter-broker replication generate cross-AZ data transfer ~2.4 TB/bulan = $48/bulan. Plus consumer cross-AZ cost.

Fix: pakai client.rack configuration untuk prefer consumer dari AZ yang sama dengan broker. Cross-AZ traffic turun ~60%.

4. NATS upgrade in-place failure

Upgrade NATS 2.10 ke 2.11, salah satu node tidak join back cluster karena state file format change minor. Cluster degrade ke R2 selama 18 menit sambil saya restore state.

Fix: pre-upgrade snapshot stream state, rolling upgrade dengan health check antar node, dan automated runbook upgrade.

5. Kafka topic deletion stuck

Topic legacy yang sudah tidak dipakai, di-mark delete. Cluster state stuck “marked for deletion” 4 hari. Ternyata delete.topic.enable=true butuh di-restart broker, bug Kafka 3.8.

Fix: rolling restart broker. Plus default config delete.topic.enable=true dari awal.

Verdict

Kafka menang untuk:

  • Workload throughput > 100k msg/s sustained
  • Compliance audit dengan log compaction + retention long-term
  • Ecosystem requirement (Kafka Connect, Schema Registry, ksqlDB)
  • Tim sudah familiar (operational knowledge transfer)

NATS Jetstream menang untuk:

  • Low-latency RPC-like + pub/sub
  • Tim kecil tanpa SRE dedicated (ops minimal)
  • Cost-conscious dengan workload moderate
  • Service-to-service messaging dengan request-reply pattern

Hybrid: untuk enterprise dengan diverse use case. Don’t force single broker untuk semua kebutuhan.

Bukan magic — saya pernah debug NATS consumer lag jam 11 malam saat upgrade gagal. Tapi untuk profile workload enterprise Indonesia: hybrid worth complexity. Lihat juga event sourcing + saga pattern untuk konteks bagaimana broker ini di-leverage di payment flow.

Ditulis oleh Reza Pradipta