2026-09-18 · 9 min
Kubernetes Resource Requests, Limits, and Rightsizing: Stop Wasting Node Capacity in 2026
Your cluster has 32 cores and 128 GB RAM across six nodes. Your workloads are consuming 4 cores and 20 GB. Yet pods are getting OOMKilled, others are throttled, and the scheduler is rejecting new pods because no node can fit them. This is not a capacity problem — it is a resource configuration problem, and it is extremely common.
I have seen this exact scenario in clusters running at a logistics company in the BSD City area — nodes apparently “full” according to the scheduler, but kubectl top nodes showing 15% actual CPU usage. The culprit was requests set too high and limits set arbitrarily, with no one having revisited them since the initial deployment eighteen months prior.
How Kubernetes actually uses requests and limits
Before touching any YAML, the mental model needs to be correct.
Requests tell the scheduler what the pod needs. A node is “full” when the sum of all pod requests exceeds the node’s allocatable capacity — not when actual utilization is high. This means you can have a node at 10% CPU utilization that refuses to schedule new pods, simply because the sum of requests is at 100%.
Limits tell the container runtime what the pod is allowed to use at runtime. Memory limits are a hard ceiling: exceed them and the kernel sends SIGKILL. CPU limits are different — they throttle the container using CFS quotas rather than killing it.
The gap between these two mechanisms causes most of the problems I see in production:
| Problem | Root Cause |
|---|---|
| OOMKilled pods | Memory limit set too low relative to actual peak usage |
| CPU throttling | CPU limit set too low; pod bursts but gets capped by CFS |
| Scheduler rejecting pods | Requests set too high; node appears full before it is |
| Wasted capacity | Requests set too high AND pods don’t use what they claimed |
The QoS class implications
Kubernetes assigns a QoS class to every pod based on how you set requests and limits. This matters because the node’s eviction order follows QoS class when it runs low on resources.
Guaranteed: requests equal limits for every container in the pod. These pods are evicted last.
Burstable: requests are lower than limits (or limits are not set). Most workloads fall here.
BestEffort: no requests or limits set at all. These are evicted first.
For critical services, Guaranteed QoS is tempting, but it means you are reserving peak capacity at all times. A service that averages 200m CPU but occasionally spikes to 800m, set with request: 800m, limit: 800m, wastes 600m of scheduled capacity 90% of the time. Burstable with accurate requests and a sensible limit headroom is usually the right call.
Setting requests: observe first, set second
The correct workflow for setting requests is to observe actual usage before writing numbers in YAML. For an existing workload, pull the 95th percentile usage over the past 14 days:
# 95th percentile CPU usage per container over 14 days
quantile_over_time(0.95,
rate(container_cpu_usage_seconds_total{
container="payment-api",
namespace="production"
}[5m])[14d:5m]
)
# 95th percentile memory working set over 14 days
quantile_over_time(0.95,
container_memory_working_set_bytes{
container="payment-api",
namespace="production"
}[14d:5m]
)
Set your CPU request at roughly the p95 value. Set memory request at p95, then add a 20–30% buffer for GC spikes, JVM metaspace growth, or whatever overhead your runtime carries. These numbers beat any “best practice” table you will find in generic documentation.
Setting limits: headroom not multiples
A pattern I have seen too often is setting limits as 2x requests without any reasoning behind the multiplier. The right approach is to think about what the limit is actually protecting against.
CPU limit protects other pods from a noisy neighbor consuming all available CPU on the node. For a well-behaved service, a limit of 2–3x the request is reasonable. For a JVM service that spikes during GC, you may need 4–5x, or you can remove the CPU limit entirely and rely on requests for scheduling fairness.
Memory limit protects the node from a leaking or runaway container. Set it at p99 usage plus headroom for burst — never at p50, because the p99 scenario is exactly the one that kills your pod in production.
# A payment-api container that averages 150m CPU / 300Mi memory
# but spikes to 400m CPU during traffic bursts and 600Mi during GC
resources:
requests:
cpu: "200m" # slightly above p95 actual usage
memory: "400Mi" # p95 usage + 30% buffer
limits:
cpu: "800m" # 4x request — allows burst without throttling GC
memory: "700Mi" # p99 + buffer; OOMKill before wasting the whole node
Detecting throttling and OOMKill candidates
Do not wait for pages to know something is wrong. Two Prometheus queries cover most of the issues:
CPU throttle ratio — anything consistently above 25% is degrading your service’s latency:
sum by (pod, container, namespace) (
rate(container_cpu_cfs_throttled_seconds_total[5m])
) /
sum by (pod, container, namespace) (
rate(container_cpu_cfs_periods_total[5m])
)
Memory pressure — pods at above 80% of their memory limit are candidates for OOMKill:
(
container_memory_working_set_bytes{container!=""}
/
on(pod, container, namespace)
kube_pod_container_resource_limits{resource="memory"}
) > 0.80
Set alerts on both. I run these in Grafana with a 10-minute evaluation window and a 5-minute for duration to avoid alerting on transient spikes. The alerts wake you up before the OOMKill loop starts.
VPA in recommendation mode
Vertical Pod Autoscaler is underused because most people look at it and immediately worry about pods being restarted automatically. Skip auto mode for now and use recommendation mode — it is entirely passive and gives you exactly what you need to rightsize.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: payment-api-vpa
namespace: production
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: payment-api
updatePolicy:
updateMode: "Off" # recommendation only — never touch running pods
resourcePolicy:
containerPolicies:
- containerName: payment-api
minAllowed:
cpu: "50m"
memory: "128Mi"
maxAllowed:
cpu: "2"
memory: "2Gi"
After a week, check the recommendations:
kubectl get vpa payment-api-vpa -n production -o json \
| jq '.status.recommendation.containerRecommendations[0]'
The output gives you lowerBound, target, and upperBound for both CPU and memory. Use target as your new request, upperBound as a starting point for the limit. Run this quarterly on all workloads. It takes less than an hour and recovers real capacity.
LimitRange: set guardrails at namespace level
Individual pod configurations drift over time. New services get deployed with no resource configuration at all — they land as BestEffort and pollute your eviction behavior. A LimitRange object sets defaults and enforces boundaries at the namespace level:
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
namespace: staging
spec:
limits:
- type: Container
default:
cpu: "500m"
memory: "256Mi"
defaultRequest:
cpu: "100m"
memory: "128Mi"
max:
cpu: "4"
memory: "4Gi"
min:
cpu: "10m"
memory: "32Mi"
Any container deployed without resource specifications in this namespace gets the default values injected automatically. The max values prevent a single misconfigured deployment from requesting 32 cores and parking a node for everyone else.
I deploy LimitRange objects to every namespace in the cluster. It is one of those configurations that feels administrative until the day someone accidentally sets requests.memory: 10000Mi in a PR and the scheduler blocks production for twenty minutes.
The rightsizing workflow
Pull it together into a repeatable process:
-
Audit current state: compare scheduled requests against actual usage using
kubectl top podsversuskube_pod_container_resource_requests. Identify workloads where requests are more than 3x actual usage. -
Deploy VPA in Off mode to all workloads that do not already have it. Wait at least 7 days for the recommender to observe meaningful traffic patterns.
-
Apply VPA recommendations to manifests. Use
targetfor requests,upperBoundfor limits. Do not blindly copy — sanity-check against known traffic spikes (month-end batch, Lebaran traffic surges, end-of-quarter). -
Set throttle and memory pressure alerts so the next round of drift does not go undetected for 18 months.
-
Add LimitRange to every namespace to prevent unset resources from slipping through.
Run steps 1 and 3 quarterly. The cost savings from reduced node count typically far exceed the time investment, and the reliability improvement — fewer OOMKill loops, less throttle-induced latency — shows up immediately in p99 response times.
Resource misconfiguration is not a Kubernetes problem. It is a feedback-loop problem: without metrics-driven guardrails, requests and limits drift from reality the moment the service is deployed. The tools exist. The gap is the habit of using them.
Ditulis oleh Reza Pradipta