Kubernetes operators struggle with CPU throttling hidden by normal utilization graphs

Set requests before limits, and learn to read CPU throttling before you tune anything. Requests decide where your pod lands and what it's guaranteed; limits just decide when it gets strangled. We spent two weeks chasing "random" latency spikes on a service that looked healthy everywhere. Turned out the CPU limit was throttling it 40% of the time. Pods never restarted, no alerts fired, utilization graphs looked fine. Bumped the request, dropped the limit to something sane, p99 went back to normal overnight. Nobody tells you a pod can be "running" and still starved.

I still can't wrap my head around cpu throttling even when pod hasn't even reached cpu requests.

CPU request is how much CPU is reserved for your pod if it needs it. The limit is a hard cap. Hit it and the kernel throttles the pod, even when the node still has spare CPU. That is the usual cause of CPU throttling on Kubernetes. It does not protect the neighboring pods. In my simple Web API test, adding a CPU limit took typical latency from 23 ms to 87 ms, (4x slower), with the limited pod throttled in half of all CFS windows, and the average CPU graph looked fine the whole time.