karawaci.kode

2026-07-01 · 7 min

Prometheus + Grafana SaaS Jakarta: Setup 2 Hari

Dua bulan lalu, klien SaaS Jakarta (logistik B2B, 6 microservice di Hetzner) datang ke saya: “Kami buta total kalau ada masalah. Logs ada tapi tidak ada metric.” Mereka pakai Datadog trial 14 hari, lalu kaget bill $186/bulan. Saya setup Prometheus + Grafana self-host dalam 2 hari kerja. Hari ini, share blueprint dan trade-off.

Konteks

  • Tim: 4 backend engineer + 1 SRE part-time (saya).
  • Service: 6 microservice (auth, order, shipping, billing, notification, admin).
  • Stack: Go (3 service), Bun (2 service), Python (1 service).
  • Traffic: ~800 RPS aggregate peak, 99k order/hari.
  • Existing: structured log ke stdout, agregasi pakai Loki (sebelumnya).

Yang dibutuhkan:

  • Metric per-service: RPS, error rate, latency p50/p95/p99
  • Metric infra: CPU, memory, disk, network per VPS
  • Metric DB: connection pool, slow query, replication lag
  • Alerting ke Slack untuk threshold violation

Setup hari 1: Prometheus

VPS Hetzner CX32 (€10/bulan, 4 vCPU, 8GB RAM, 80GB disk). Dedicated untuk monitoring stack.

Docker Compose:

services:
  prometheus:
    image: prom/prometheus:v3.0.1
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--storage.tsdb.retention.size=20GB'
      - '--web.enable-lifecycle'
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:v1.8.2
    network_mode: host
    pid: host
    volumes:
      - /:/host:ro,rslave
    command:
      - '--path.rootfs=/host'
    restart: unless-stopped

volumes:
  prom-data:

Prometheus config (prometheus.yml):

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'jakarta-prod'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets:
        - 'app1.internal:9100'
        - 'app2.internal:9100'
        - 'app3.internal:9100'
        - 'db1.internal:9100'

  - job_name: 'services'
    static_configs:
      - targets:
        - 'app1.internal:3001'  # auth
        - 'app1.internal:3002'  # order
        - 'app2.internal:3003'  # shipping
        - 'app2.internal:3004'  # billing
        - 'app3.internal:3005'  # notification
        - 'app3.internal:3006'  # admin
    metrics_path: /metrics

  - job_name: 'postgres'
    static_configs:
      - targets: ['db1.internal:9187']

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - 'alerts/*.yml'

Network: semua scrape via WireGuard mesh antar VPS (private). Tidak expose port 9100/9187 ke internet.

Instrumentation per service

Go service (auth, order, shipping)

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    httpRequests = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total HTTP requests",
        },
        []string{"method", "path", "status"},
    )
    
    httpDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP request duration",
            Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5},
        },
        []string{"method", "path"},
    )
)

func init() {
    prometheus.MustRegister(httpRequests, httpDuration)
}

func MetricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        wrapped := &statusRecorder{ResponseWriter: w, status: 200}
        next.ServeHTTP(wrapped, r)
        
        path := normalizePath(r.URL.Path) // /users/123 -> /users/:id
        httpRequests.WithLabelValues(r.Method, path, strconv.Itoa(wrapped.status)).Inc()
        httpDuration.WithLabelValues(r.Method, path).Observe(time.Since(start).Seconds())
    })
}

normalizePath penting — tanpa itu, label cardinality explode (jutaan unique path dengan ID).

Bun service (billing, notification)

import { Hono } from 'hono';
import { register, Counter, Histogram, collectDefaultMetrics } from 'prom-client';

collectDefaultMetrics({ register });

const httpRequests = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'path', 'status'],
});

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'path'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

app.use('*', async (c, next) => {
  const start = performance.now();
  await next();
  const duration = (performance.now() - start) / 1000;
  const path = normalizePath(c.req.path);
  httpRequests.inc({ method: c.req.method, path, status: c.res.status });
  httpDuration.observe({ method: c.req.method, path }, duration);
});

app.get('/metrics', async (c) => {
  c.header('Content-Type', register.contentType);
  return c.body(await register.metrics());
});

Hari 2: Grafana + dashboard

grafana:
  image: grafana/grafana:11.4.0
  ports: ["3000:3000"]
  environment:
    GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PWD}
    GF_USERS_ALLOW_SIGN_UP: 'false'
    GF_AUTH_GOOGLE_ENABLED: 'true'
    GF_AUTH_GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID}
    GF_AUTH_GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET}
    GF_AUTH_GOOGLE_ALLOWED_DOMAINS: 'kami.co.id'
  volumes:
    - grafana-data:/var/lib/grafana

9 dashboard yang saya buat:

  1. Service Overview — RPS, error rate, latency per service.
  2. Infrastructure — CPU, memory, disk, network per VPS.
  3. Postgres — connection, query rate, slow query, replication lag.
  4. Business KPI — order/hari, revenue, signup rate.
  5. Per-service deep dive (×6, satu per service).

Query Prometheus untuk error rate (PromQL):

sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
/
sum(rate(http_requests_total[5m])) by (job)
* 100

Untuk latency p95:

histogram_quantile(0.95, 
  sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
)

Alerting ke Slack

alerts/services.yml:

groups:
  - name: services
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)
          > 0.02
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: "{{ $labels.job }} error rate {{ $value | humanizePercentage }}"

      - alert: HighLatencyP95
        expr: |
          histogram_quantile(0.95, 
            sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
          ) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency > 500ms on {{ $labels.job }}"

      - alert: PostgresReplicationLag
        expr: pg_replication_lag_seconds > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Postgres replica lag {{ $value }}s"

Alertmanager → Slack webhook. Lihat config alertmanager.yml:

route:
  receiver: 'slack-default'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="critical"]
      receiver: 'slack-critical'
      repeat_interval: 30m

receivers:
  - name: 'slack-default'
    slack_configs:
      - api_url: '${SLACK_WEBHOOK_URL}'
        channel: '#ops-alerts'
  - name: 'slack-critical'
    slack_configs:
      - api_url: '${SLACK_CRITICAL_URL}'
        channel: '#ops-critical'

Hasil setelah 2 bulan

  • 47 metrik custom ditrack (RPS, latency, error, business KPI).
  • 9 dashboard dipakai aktif (saya track view count via Grafana annotation).
  • Storage: 14GB untuk 30 hari retention, 6 service + node exporter.
  • Alert sent: 23 (8 actionable, 12 noise filtered out, 3 critical incident).
  • Mean time to detect (MTTD): dari estimasi 20-30 menit (via customer complaint) ke 90 detik (via alert).
  • Cost: €10/bulan VPS, vs Datadog $186/bulan. Saving Rp 2.6 juta/bulan.

Yang break

1. Label cardinality explosion

Minggu pertama, salah satu service punya label user_id di metric. 12k tenant × 8 endpoint = 96k unique series. Prometheus memory naik ke 6GB, scrape interval melar ke 40+ detik.

Fix: drop label user_id, ganti dengan label tenant_tier (3 nilai). Prometheus memory turun ke 1.2GB.

Pelajaran: jangan label dengan high-cardinality value (user ID, request ID, timestamp). Pakai bucket atau aggregate.

2. Alert fatigue

Minggu kedua, 47 alert sent dalam 3 hari. 38 noise (transient spike, false positive). Tim mulai ignore #ops-alerts.

Fix: tuning threshold + for: 5m (require sustained, bukan single sample). Plus saya tambahkan inhibit rule: alert “HighErrorRate” suppress alert “HighLatencyP95” untuk service sama (root cause kemungkinan sama).

Alert rate setelah tuning: 3-5/minggu, mostly actionable.

3. Scrape timeout di service Python

Service notification (Python + FastAPI) kadang timeout di scrape (>10s). Investigasi: GIL contention saat compute heavy report.

Fix: pisahkan endpoint /metrics ke thread pool terpisah pakai prometheus_client.start_http_server() di port berbeda. Decouple metric server dari main API.

4. Disk fill di hari ke-21

Retention saya set 30 hari + 20GB cap. Hari ke-21, disk 80GB hampir penuh karena ada tabel yang label-nya pelan-pelan grow (status code 500 kadang muncul dengan path baru tiap deploy karena bug routing). Prometheus pakai 19GB, retention belum trigger.

Fix: monitor prometheus_tsdb_storage_blocks_bytes dan alert kalau > 80% target. Plus audit metric cardinality bulanan.

Verdict

Prometheus + Grafana self-host untuk SaaS ukuran SMB Indonesia: setup 2 hari, cost €10/bulan, saving signifikan vs SaaS APM.

Trade-off: Anda jadi operator. Kalau Prometheus down, Anda yang restart. Kalau cardinality explode, Anda yang debug. Untuk tim dengan 1 orang yang willing handle ops: worth. Untuk tim tanpa SRE bandwidth: bayar Datadog atau Grafana Cloud.

Saya pakai stack ini untuk 3 klien sekarang. Tidak satupun yang regret. Untuk konteks observability lebih dalam ke logs lihat juga LogRocket vs Sentry vs Datadog.

Ditulis oleh Reza Pradipta