You're reading one of the top pains on Grafana — a free sample of the catalog. Create a free account to browse all Shopify & Atlassian pains and the basic Chrome catalog.
Kubernetes operators face CPU throttling and OOM from misconfigured resource requests and
.21
Demand — how loud & how many (score)
Willing to payImplicit
Momentum (7d)▼ 75.5%
Cluster10 posts · 9 people
MarketplaceGrafana
Updated3d ago
The story
The pain
We're seeing production issues because CPU limits cause throttling even when the node has spare capacity, and memory limits above requests lead to OOM kills for neighboring pods. Default JVM heap sizing and missing resource requests on batch workloads make things worse, causing cascading failures that look like application bugs. Setting CPU requests to average usage, removing CPU limits, and matching memory requests to limits helps, but it's a constant tuning effort.
Evidence
What people said · 10 complaints, 9 people
CPU request is how much CPU is reserved for your pod if it needs it. The limit is a hard cap. Hit it and the kernel throttles the pod, even when the node still has spare CPU. That is the usual cause of CPU throttling on Kubernetes. It does not protect the neighboring pods. In my simple Web API test, adding a CPU limit took typical latency from 23 ms to 87 ms, (4x slower), with the limited pod throttled in half of all CFS windows, and the average CPU graph looked fine the whole time.
One thing missing here that cost us a night. If you never set heap flag at all, JVM takes 25 percent of the container limit, so 1.5Gi container runs with about 384Mi heap and you get OOM while memory graph still looks half empty. Second part is worse, default JVM does not exit on OOM, it limps, so orchestrator never restarts it and in flight work just hangs. For us that surfaced as storm of database connection errors six seconds later, and we spent the night staring at a completely healthy database. Now we set heap fraction explicitly, cap metaspace, and add ExitOnOutOfMemoryError so the pod dies honestly and gets restarted.
The one that bit us hardest: no resource requests on a batch workload. Scheduler treated the pods as free, packed them onto a node already running our ingress controller, and the OOM killer got to pick a victim. It picked ingress. Whole cluster looked down from the outside while every pod reported Running.
CPU request too high (200–500m). My services barely use 20m steady-state. This is pure waste and it's what's driving my node count. CPU limit only a bit above request (800m–1000m). Causing two problems: (1) Java services are slow / crash-loop at startup , because the JVM needs a big CPU burst to boot (JIT + class loading + Spring context); (2) occasional slow requests from CPU throttling . Memory request is fine (600Mi–1000Mi). Matches what monitoring shows. Memory limit higher than request — and I now think this is a mistake. Unlike CPU, memory isn't reversible: once a pod grabs memory above its request it keeps it (a JVM especially never gives heap back). So the burst headroom just risks OOMKills for other pods on the node, with no real upside.
mem req should always equal mem limit, unless your willing to trade stability cpu limit most cases should not be specified unless your app does dynamic thread/process allocation (common in Go and Java) . In those cases limit should match params your passing to Go/Java of what their cpu limit is Set cpu req to your average usage RE: with no cpu limit will that starve other containers ? No it won't! As long as all other pods have cpu req set, k8s guarantees the req amount is available to each pod - so effectively your neighbour pods on the same node req values act as limits on your pod.
The latter “mistake” I see commonly done in cluster autoscaling environments because they all make scaling decisions by cumulative resource requests. Well, you get a Pod that decides to have a bit more work than it typically has at its base resource usage, or a handful of Pods maybe, and they can easily OOM a node and crash kubelet. Now you’re dealing with a production incident for dozens of Pods running on that node. So people set Pod requests to a similar (or same) value as limits so that they aren’t dealing with Pagerduty alerts in the middle of the night about needing to do an AWS ASG instance refresh to flush out the dead node.
Export the pain — problem, evidence and market signals — as Markdown, PDF or JSON. The Markdown export ships with a built-in briefing you can hand straight to a developer or an AI agent.