CPU Throttling and OOMKills: Getting Kubernetes Resource Limits Right

The Symptom That Doesn't Match the Dashboard

A common support call goes like this: "The cluster has plenty of capacity, nodes are at 40% CPU, but our API is slow and pods keep restarting." The dashboard looks fine, so the team assumes the application is at fault. In practice, the problem is almost always misconfigured resource requests and limits.

Kubernetes resource management is deceptively simple to configure and surprisingly easy to get wrong. The two failure modes you will see in production are CPU throttling and OOMKills. Both are invisible unless you know exactly where to look.

Requests vs. Limits: A Quick but Necessary Refresher

Requests are what the scheduler uses to decide where a pod fits. A pod requesting 500m CPU and 512Mi memory will only be placed on a node that has that much uncommitted. Requests also define your guaranteed share under contention.

Limits are the hard ceiling enforced at runtime. CPU limits are enforced by the Linux CFS scheduler through throttling. Memory limits are enforced by the kernel OOM killer.

The critical detail most teams miss: CPU and memory limits behave completely differently. Exceeding a CPU limit slows you down. Exceeding a memory limit kills you.

CPU Throttling: The Silent Latency Tax

When a container hits its CPU limit, the kernel does not let it use more CPU even if the node is idle. Instead it throttles the process: the container is paused until the next CFS period (default 100ms) begins.

This produces latency spikes that look random. A request that normally takes 20ms occasionally takes 120ms because the worker thread was paused mid-request. Your average latency looks acceptable, but p99 is terrible, and no single component appears overloaded.

The metric to watch is container_cpu_cfs_throttled_periods_total versus container_cpu_cfs_periods_total. If a meaningful percentage of periods are throttled, your limit is too low for the workload's bursty behavior.

The counterintuitive fix is often to raise or remove CPU limits while keeping requests accurate. Requests still guarantee your fair share under contention, but removing the hard ceiling lets latency-sensitive workloads burst into spare capacity instead of being artificially paused. Reserve strict CPU limits for noisy-neighbor isolation, not as a default everywhere.

OOMKills: When the Kernel Wins

Memory is non-compressible. If a container tries to allocate beyond its limit, the kernel kills the process with signal 9. You will see the container restart with OOMKilled as the reason, and a corresponding event.

The nasty part is that an OOMKill is abrupt. There is no graceful shutdown, no flushing of in-flight work. For a stateful workload or a queue consumer, this can mean lost or duplicated work.

Memory limits should be set, and they should be set with headroom. Profile actual usage under realistic load, look at the steady-state plus peak, and add a buffer. JVM, Node.js, and Go workloads each have their own memory behavior; for the JVM in particular, make sure heap settings are aware of the cgroup limit, or the runtime will size itself for the whole node and get killed instantly.

The QoS Classes Nobody Configures On Purpose

Kubernetes assigns every pod a Quality of Service class based on how you set requests and limits:

• Guaranteed: requests equal limits for every container. First to survive node pressure.
• Burstable: requests set, limits higher or absent. The pragmatic default for most workloads.
• BestEffort: nothing set at all. First to be evicted when a node runs out of memory.

The danger is accidental BestEffort pods. A team ships a manifest with no resource block, and that pod becomes the sacrificial victim during node memory pressure. Critical services should never be BestEffort by accident.

A Practical Starting Policy

There is no universal number, but there is a sane default approach:

1. Always set memory requests and limits, with the limit close to observed peak plus a safety margin. Treat OOMKills as bugs to investigate, not noise to ignore.
2. Always set CPU requests based on realistic average usage so the scheduler packs nodes correctly.
3. Be cautious with CPU limits. For latency-sensitive services, consider leaving them off and relying on requests for fairness. For batch or untrusted workloads, set them.
4. Avoid massive overcommit on requests that leaves nodes unable to honor guaranteed shares under load.

Make It Observable, Then Enforce It

Dashboards based on average node CPU hide all of this. Add panels for per-container throttling ratio and OOMKill counts, and alert on them directly. Once you can see throttling and kills, the tuning becomes data-driven instead of guesswork.

For long-term hygiene, tools like the Vertical Pod Autoscaler in recommendation mode can surface where requests and limits drift from reality. Combine that with admission policies that reject pods missing resource blocks, and you eliminate the accidental BestEffort class entirely.

Resource limits are not a one-time setting. As traffic patterns and code change, so should your numbers. The teams that run Kubernetes calmly are the ones treating these values as living configuration backed by real metrics.