Redis Cluster vs Sentinel: SaaS Jakarta Skala 100k Op/s

Setahun lalu, SaaS Jakarta yang saya support (CRM + marketing automation, ~22 juta MAU, 1.800 RPS API + heavy background job) hit ceiling Redis Sentinel: memory 56GB/64GB single node, throughput 95k op/s puncak. Replication lag mulai naik ke 200-400ms saat peak hour. Tim mau decide: scale-up Sentinel atau migrate ke Cluster.

Pilih Cluster. Share keputusan, migration plan, dan operational lessons setelah 9 bulan production.

Konteks workload

• App: SaaS CRM + marketing automation B2B Indonesia.
• Tenant: ~8.400 perusahaan klien.
• Use case Redis:
  • Session store (TTL 24 jam)
  • Cache campaign send status (TTL 7 hari)
  • Rate limiter per-tenant
  • Queue (BullMQ-like) untuk background job
  • Real-time presence (chat-like feature)
• Skala: 56GB dataset, 95k op/s peak (read 75%, write 25%).
• SLA: 99.95% available, p99 < 5ms untuk session lookup.

Setup awal: Redis Sentinel dengan 1 master + 2 replica, total 3 node Hetzner CCX33 (8 vCPU, 32GB RAM, NVMe).

Trigger migrate

Tiga sinyal:

1. Memory pressure: 56GB/32GB RAM = swap. P99 latency naik dari 1.2ms ke 8ms.
2. Single-node throughput ceiling: peak 95k op/s, plateau di sini meskipun CPU baru 65%.
3. Replication lag: 200-400ms peak hour, risiko data loss saat failover.

Saya tidak langsung pilih Cluster. Evaluate dulu:

Option	Pro	Contra
Scale-up Sentinel (RAM 96GB)	Simple, no client code change	Sembuh sementara, throughput tetap single-node
Multiple Sentinel cluster (per-use-case)	Isolate workload, simple sharding logic	Complexity ops, no atomic cross-use-case
Redis Cluster	Horizontal scale memory + throughput	Multi-key constraint, client library compatibility
ElastiCache + sharding	Managed	Vendor lock-in, cost 3x self-host

Pilih Cluster karena: roadmap workload growth 40% YoY, single-node tidak akan cukup 12 bulan ke depan.

Architecture: Redis Cluster 6-shard

Topology:

• 6 master shard + 6 replica = 12 node
• Hetzner CCX33 (8 vCPU, 32GB RAM) per node
• 2 region: Jakarta (5 master + 5 replica) + Bandung (1 master + 1 replica untuk geo-spread)
• WireGuard mesh antar node untuk security

Total memory: 192GB (6 × 32GB), dengan headroom 60% utilization target = 115GB usable.

Config (redis.conf):

cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-require-full-coverage no
cluster-replica-validity-factor 10

maxmemory 24gb
maxmemory-policy allkeys-lru

appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 4gb

repl-diskless-sync yes
repl-diskless-sync-delay 5

protected-mode yes
requirepass xxx
masterauth xxx

cluster-require-full-coverage no penting untuk Indonesia network reality: kalau 1 shard partition, sisanya tetap serving. Trade-off: keys di shard mati return error, bukan whole cluster down.

Key partitioning strategy

Default cluster hash: CRC16(key) mod 16384 slot. Tapi multi-key ops butuh hash tag.

Pattern 1: session

Key: session:{user_id}:{token} → hash by user_id:

session:{u_12345}:abcdef123 → slot from "u_12345"
session:{u_12345}:fedcba456 → same slot

Pakai {user_id} curly brace = hash tag. Multi-session per user di shard sama → MGET feasible.

Pattern 2: campaign cache

Key: campaign:{tenant_id}:{campaign_id}:status → hash by tenant_id:

campaign:{t_acme}:c_001:status
campaign:{t_acme}:c_002:status

Per-tenant queries co-locate. Cross-tenant query mandatory split.

Pattern 3: rate limiter

Key: rl:{tenant_id}:{endpoint} → same shard per tenant per endpoint:

rl:{t_acme}:GET-/api/contacts

INCR atomic at shard. Cross-tenant aggregate via separate index.

Pattern 4: queue (problematic)

BullMQ pre-Cluster era pakai banyak Lua script + multi-key. Saat migrate ke Cluster, BullMQ butuh konfig prefix per queue + hash tag:

new Queue('email-blast', {
  connection: clusterRedis,
  prefix: '{email-blast}', // all keys for this queue same slot
});

Side effect: one queue = one shard. Scale across shard butuh banyak queue (e.g., {email-blast-1}, {email-blast-2}, … sharded by tenant hash di app level).

Migration: 6 minggu

Phase 1 (1 minggu): provision cluster baru, dual-write mode di app.

Phase 2 (2 minggu): backfill data lama dari Sentinel ke Cluster:

# Per shard, pakai redis-cli --cluster import
redis-cli --cluster import cluster-host:6379 \
  --cluster-from sentinel-master:6379 \
  --cluster-copy \
  --cluster-replace

Untuk live data (session, cache), bukan import semua — biar TTL expire, dual-write isi cache baru naturally. Untuk persistent data (queue), explicit migrate.

Phase 3 (2 minggu): traffic ramp-up. Pakai feature flag (lihat GrowthBook setup):

• 5% tenant read dari Cluster
• 25% tenant
• 100% tenant

Phase 4 (1 minggu): decommission Sentinel.

Performance

Metric	Sentinel	Cluster (6-shard)
Throughput sustainable	95k op/s	320k op/s
Memory usable	32GB (single shard)	115GB (6 shard)
Latency p50	1.1ms	0.9ms
Latency p99	8ms	2.4ms
Failover time	22 detik	11 detik
Cost monthly	€120 (3 node)	€480 (12 node)

Latency p99 turun karena per-shard memory pressure hilang (no swap). Cost naik 4x tapi memory headroom 3.5x dan throughput 3.3x.

Failover

Cluster failover via gossip protocol, no Sentinel needed:

• cluster-node-timeout 5000 = 5 detik gossip timeout
• Replica promote ke master: ~3-5 detik
• Client library redirect via MOVED error: ~1-3 detik
• Total observed: 11 detik p95 (vs 22 detik Sentinel)

Failover test setiap kuartal (chaos engineering):

# Kill master shard 3
ssh shard-3-master "sudo systemctl stop redis"

# Monitor cluster status from another shard
redis-cli -h shard-1-master cluster info | grep state
# cluster_state:ok (with shard 3 master gone)

# Replica promotion
redis-cli -h shard-1-master cluster nodes | grep "shard-3"
# Look for myself master vs slave transition

Yang break

1. CROSSSLOT error di code legacy

Setelah cutover, ~3% request error dengan CROSSSLOT Keys in request don't hash to the same slot. Code yang panggil MGET multiple key tanpa hash tag.

Fix: audit code, ganti MGET dengan pipeline-of-GET (compatible cluster), atau add hash tag. 4 hari engineering.

2. BullMQ delayed job stuck

BullMQ pakai Lua script untuk atomic enqueue + dequeue. Saat migrate ke Cluster, salah satu script hardcoded key tanpa hash tag, gagal di Cluster.

Fix: upgrade BullMQ ke v5.13+ yang punya Cluster-aware Lua script. Plus prefix kuratif:

new Queue('jobs', { prefix: '{bull}' });

3. Slot migration manual saat scale

Bulan ke-6, traffic naik ~30%, tambah 2 shard (8 total). Slot migration redis-cli --cluster reshard butuh manual approval dan monitor. Selama 4 jam migration, latency p99 naik ke 12ms untuk subset key di slot yang sedang migrate.

Fix forward: pakai Redis Enterprise atau ElastiCache yang automate. Untuk self-host: tetap manual, schedule di off-peak window.

4. Replica lag during big AOF rewrite

Saat AOF rewrite (4-5x sehari di shard busy), CPU spike, replication lag ke replica naik ke 2-3 detik. Read-from-replica yang stale.

Fix:

• auto-aof-rewrite-min-size 4gb (lebih besar, less frequent rewrite)
• Off-peak schedule rewrite via cron + BGREWRITEAOF
• Application-level “stale read tolerance” flag per use case

5. Cluster split-brain saat WireGuard down

Insiden bulan ke-3: WireGuard config typo saat update, mesh between Jakarta-Bandung partition selama 8 menit. Cluster gossip detect, tapi Bandung shard isolated ditandai “fail”.

Karena cluster-require-full-coverage no, Jakarta side tetap serving 5/6 shard. 1/6 keyspace (di Bandung) return error.

Fix: pasca insiden, monitoring WireGuard tunnel health + alert kalau handshake gagal > 30 detik. Plus rehearsal disaster recovery procedure tiap kuartal.

6. Memory fragmentation

Setelah 4 bulan running, beberapa shard punya fragmentation ratio 1.6 (allocated vs RSS). RSS 28GB tapi data 17GB. Waste.

Fix: activedefrag yes plus active-defrag-threshold-lower 10 (start at 10% fragmentation). CPU overhead ~3-5%, fragmentation turun ke 1.15.

Kapan tetap pilih Sentinel

1. Memory < 50GB total dataset: single-node cukup, Cluster overhead tidak worth.
2. Throughput < 80k op/s: single shard masih ada headroom.
3. Heavy multi-key operation (lots of MGET, MSET, transaction): hash tag constraint bisa painful.
4. Tim kecil tanpa Redis SRE knowledge: Sentinel operational simpler.

Kapan Cluster mandatory

1. Memory > 60GB (single-node max ~96GB practical, butuh headroom).
2. Throughput > 100k op/s sustained.
3. Geographic distribution untuk DR.
4. Workload natural shard-able (per-tenant, per-user).

Verdict

Untuk SaaS Indonesia di skala 100k+ op/s atau memory > 60GB: Redis Cluster worth complexity. Operasional ops butuh disiplin (hash tag discipline, monitor per-shard, slot migration window), tapi return scalability dan failover lebih baik.

Untuk skala lebih kecil: stick dengan Sentinel. Don’t over-engineer.

Bukan magic. Migrasi 6 minggu, debug CROSSSLOT 4 hari, insiden split-brain 8 menit. Tapi 9 bulan ini cluster 99.97% available + handle 3x traffic dari Sentinel baseline. Konteks rate limiter pattern di caching strategy Redis CDN juga membantu.