For the love of god stop using CPU limits in Kubernetes

Platform engineering · analysis We tested the same app with and without a CPU limit. Same code, same CPU request, same load. The only difference was the limit. Here is what we measured - and how to check it on your own cluster. - Remove CPU limits. They freeze your apps many times per second, even when the node has free CPU. - Keep CPU requests. Requests are the real protection. They guarantee every app its share. - Keep memory limits. Memory is different. A memory limit still protects the node. - Faster: tail latency holds up under traffic peaks instead of collapsing, and CPU-bound startup work finishes about 2x sooner (sections 5, 10). - Less hardware: most clusters reserve far more CPU than they ever use at peak. Right-sizing requests after dropping limits lets a meaningful share of nodes go (section 10). - Cheaper: a worked, illustrative cost model puts this in the tens of thousands of dollars per year per cluster (section 10) - plug in your own prices to get a real number. In this analysis: 1. Requests vs limits · 2. How throttling works · 3. CFS fair sharing · 4. The worst case · 5. What we measured · 6. Noisy neighbor · 7. When it breaks · 8. One step for .NET · 9. The plan · 10. Outcomes · 11. Q&A · 12. Glossary Going deeper. This page is the argument. Each claim in it is backed by a longer document, linked inline as you read and collected here: | docs/01-theory.md | The cgroup mechanics: cpu.max vs cpu.weight , multi-thread quota burn, and the one configuration where a CPU limit genuinely helps. | | docs/02-dotnet.md | What a CPU limit does to a .NET runtime: ProcessorCount , ThreadPool starvation, GC heap count, and how a CPU limit causes a memory kill. | | docs/03-postgres.md | Postgres specifics: why it ignores your quota when sizing itself, and which effects this PoC measured versus merely inferred. | | docs/04-objections.md | Every counter-argument we met, answered - noisy neighbors, HPA/KEDA, QoS class, multi-tenancy, and when limits are right. | | docs/05-cost.md | The full cost model, its assumptions, and the memory floor that stops freed cores becoming freed money. | | docs/06-rollout.md | The staged rollout: six steps, what gates what, and the one-line rollback. | | results/ | Raw output from the benchmark run, plus the per-scenario reports the numbers below come from. | | CPU request | CPU limit | | |---|---|---| | What it is | A guaranteed slice of CPU | A hard ceiling, a wall | | Protects other apps? | Yes. CPU is shared by request size. | No. It only blocks its own app. | | Idle CPU on the node | App may borrow it for free | Wasted. The limit blocks it. | Same app twice. The only difference is access to the node's idle CPU. The request (solid) is identical in both rows - borrowing idle CPU takes nothing from anyone. Deeper: these are two different cgroup files, not two settings of one knob. docs/01-theory.md walks through cpu.max (the limit) andcpu.weight (the request) and what each one actually controls. The kernel (the core part of the operating system) enforces limits in windows of 100 milliseconds. A 500m limit (500 millicores, half a core) means: 50 ms of CPU time per window. When the budget runs out, the kernel freezes the whole app until the next window. This is throttling. A typical .NET service runs many threads: HTTP handlers, background consumers, and the GC (the garbage collector). All threads share one budget. Our test node has 4 cores, so at most 4 threads can run at the same instant. 8 busy threads still use up the 50 ms budget in about 12.5 ms of real time: The freezes repeat up to 10 times per second. The next chart shows why your dashboards never see them. Every orange second the app hit its limit and the kernel froze it. The 1-minute average never comes close to the limit, so every graph looks healthy. This is how a container can be throttled all day while its dashboard stays green. If you only ever look at averaged CPU, you cannot see this - check container_cpu_cfs_throttled_periods_total instead. We proved this. We sent load that needs only 320m on average to an app with a 500m limit. The average never touched the limit. The app still stalled: Same app, same load (8 parallel tasks of 5 ms, 8 requests per second). The limit made the slow requests 2.4x slower, while average CPU stayed well under the limit. Deeper: why parallelism makes this so much worse than it looks - 16 threads burn a 300m quota in under 2 ms of wall time - and why the result is a stall rather than a slowdown, which is what destroys your tail latency while the average looks fine. Linux has a built-in referee: CFS, the Completely Fair Scheduler. Every pod has a weight. Kubernetes sets the weight from the pod's CPU request. The rule is simple: when the node is fully busy, pods share the CPU based on their weights. When a pod is idle, it stops using its share. Other pods can use that share instead. The idle pod gets its share back as soon as it has work again. Three apps on one node, weights from requests: A = 100m, B = 200m, C = 700m. This referee runs on every Linux server, always. It needs no CPU limit to work. A limit adds only one thing extra: the freezes from section 2. Deeper: the full argument that a limit provides no protection a request does not already provide, plus the one real exception: Guaranteed QoS with the static CPU manager, where the "limit" is pinning cores rather than throttling quota. No. This fear treats CPU and memory as the same problem. Here is the worst possible CPU moment on a node: A 4-core node, fully busy. CFS splits CPU by request weight. Each pod drops back to its guaranteed request share - apps get slower, nothing breaks. The scheduler already makes sure the requests of all pods fit inside the node. Three protections make node failure a non-issue, all independent of CPU limits: | 1. Reserved system CPU | The OS and kubelet (the agent that runs and watches pods on a node) have their own reserved CPU slice, outside the pod pool. A busy pod cannot touch it. The node always stays responsive. | | 2. Scheduler math | Pods are placed by requests, and total requests never exceed node capacity. Full contention (every pod wanting CPU at once) still means everyone gets their request. That is the same guarantee limits give, without wasting idle CPU. | | 3. CPU is compressible | CPU can be compressed: too little of it just makes an app slower. It finishes the delayed work later. It does not crash. Memory is different: too little memory kills the app. That is exactly why memory limits stay. | | Resource | Not enough of it means | So the rule is | |---|---|---| | CPU | App runs slower for a moment | Remove the limit, keep the request | | Memory | App crashes (OOM kill, out-of-memory kill) | Keep the limit | And in most clusters, node CPU utilization sits far below capacity - often in the single or low double digits. The "everyone at 100%" moment is largely theoretical, while the real, daily problem is the opposite one: idle CPU that throttled apps are not allowed to touch. If a node ever does get fully busy, the two protections above still hold: reserved system CPU keeps the node and kubelet responsive, and CFS weight-based sharing still guarantees every pod its requested share. A node-level CPU dashboard would show sustained high usage well before this becomes a concern. This is a fair objection, and the concern behind it is real: if every application shares one namespace with no ResourceQuota and no LimitRange, nothing forces a team to set accurate requests. A common interim step is to set something like limits.cpu: 1 as a "circuit breaker rather than a ceiling" - high enough that throttling stops in practice. Setting a high limit is a fine emergency fix, and it does end throttling for the service it is applied to. But as a safety mechanism for the pods around that service, a CPU limit does not do what it looks like it does. | The worry | What actually protects you | |---|---| | "An unlimited pod has nothing protecting the workloads around it." | A pod's own limit never protects its neighbors. It only restricts itself. Neighbor A is protected from greedy pod B by A's own request, which sets A's weight in the kernel. Putting a limit on B does nothing for A. | | "Requests are not mandatory, so they cannot be trusted." | True, and that is the gap worth closing. But a pod with a missing or tiny request is the victim, not the aggressor: it gets the smallest weight and is the first to be squeezed. Adding CPU limits everywhere does not give it protection it lacks. | | "We need a circuit breaker until the guardrails exist." | The scheduler already is one. It only places a pod if its request fits the node, so total requests never exceed capacity. That is what guarantees everyone can get their share at the same time. Limits play no part in it. | The dilemma with "circuit breaker" limits. A limit set high enough never to trip (30x real use) protects nobody, because it never does anything. A limit low enough to actually trip only harms the pod it is attached to, never the neighbor it was meant to protect. It cannot be both a safety mechanism and harmless. One measurement warning on sizing a limit as "30x what the service uses". You cannot size a limit from usage measured while a limit was throttling that same workload. A heavily throttled pod's recorded usage is what the limit allowed, not what the app wanted. Sizing from suppressed usage just builds the next too-small limit. So what is actually needed? Exactly what the objection names: a LimitRange giving every pod a default request, and a ResourceQuota capping total requests per namespace. Both work on requests, not limits, and both can be applied to an existing shared namespace as-is - no namespace redesign has to come first. In practice the gap is often smaller than it looks, since a shared Helm chart typically sets a request for every service it templates. To be clear about sequencing, because it is easy to overstate: the LimitRange is