Stop Guessing Your Node Groups: A Deep Dive Into Karpenter on EKS

If you have run a production EKS cluster for more than a few months, you already know the ritual. You size your node groups based on a rough estimate of peak load. You pad that estimate, because nobody wants a 3 a.m. page about pending pods. Then you watch the bill creep upward while kubectl top nodes quietly tells you that half your fleet is idle. That is not a scheduling problem. It is a forecasting problem, and Kubernetes was never designed to solve forecasting on its own. This post walks through why the traditional model hits a ceiling, what Karpenter actually does differently at the API level, and the specific things that bite teams in production. By the end you should be able to reason about a NodePool config rather than copy one.

Part 1: Understanding the Ceiling You Are Hitting

Before Karpenter, autoscaling on EKS almost always meant Cluster Autoscaler plus Auto Scaling Groups. The loop looked like this:

1. You define managed node groups, each backed by an ASG tied to a specific instance type or a narrow family.
2. Cluster Autoscaler watches the API server for pods in Pending with a FailedScheduling event.
3. It simulates: "if I added one node from node group X, would this pod schedule?"
4. If yes, it increases that ASG's desired capacity by one.
5. The ASG launches an instance, the kubelet registers, the scheduler binds the pod.

Step 3 is where the whole design shows its age. Cluster Autoscaler can only simulate against node groups that already exist. It cannot invent a node shape. If your node group is built on m5.xlarge (4 vCPU, 16 GiB) and a burst of pods each request 2 vCPU and 14 GiB, every node you add fits exactly one pod and strands 2 vCPU. Cluster Autoscaler will happily do this all day, because from its perspective the simulation succeeded.

There is a second, subtler cost. Cluster Autoscaler requires every node in a group to be homogeneous for its simulation to be valid. If you mix instance types inside one ASG, the autoscaler's estimate of "what a node from this group provides" becomes wrong, and it either over-provisions or refuses to scale. So you split groups. One per instance family. Then one per capacity type, because Spot and On-Demand need different labels and taints. Then one per AZ, because you need zonal balance for topology spread constraints. Three families times two capacity types times three AZs is eighteen node groups, each with its own launch template, its own AMI version, and its own upgrade cycle. That matrix goes stale the moment your workload mix changes, and it is maintained by a human who has other things to do.

The core insight: the node group model forces you to make instance decisions at config-authoring time, when you know the least. The demand signal arrives at scheduling time, when you know the most. Everything painful about the old model flows from that gap.

Part 2: What Karpenter Actually Does

Karpenter deletes the node group as the unit of scaling decisions. Instead of "which existing group should I grow," it asks: "given these specific pending pods, what is the best possible instance to launch right now?"

Here is the real loop:

1. Watch for unschedulable pods. Karpenter watches for pods the kube-scheduler could not place. Same trigger as Cluster Autoscaler so far.
2. Batch them. This part matters and is frequently missed. Karpenter does not react to one pod at a time. It waits a short batching window (starting at 1 second, extending up to 10 seconds as more pods arrive) and collects everything pending. This is why a Deployment scaled from 1 to 50 replicas produces a sensible handful of large nodes rather than 50 separate provisioning decisions.
3. Bin-pack in memory. Karpenter takes that batch and solves a packing problem. It reads each pod's resource requests, node selectors, node affinity, taints and tolerations, topology spread constraints, and pod affinity rules, then computes which combination of instance types could host them. Crucially, it evaluates many instance types at once, not one.
4. Call EC2 Fleet. Karpenter passes a ranked list of viable instance types to the EC2 Fleet API with a lowest-price allocation strategy. EC2 picks the cheapest one that has capacity. This is why Karpenter handles Spot capacity crunches gracefully: if c6g.4xlarge Spot is exhausted in us-east-1a, Fleet just falls through to the next candidate instead of failing the whole request.
5. Bind pods directly. Karpenter pre-binds the pending pods to the node it is about to create, before the node even registers. This skips a scheduler round trip and cuts time-to-ready noticeably.

The thing to internalize: Karpenter makes the instance decision at scheduling time, using real demand. That single change is what collapses the eighteen-node-group matrix into one NodePool.

Part 3: Consolidation, Where the Money Actually Is

Provisioning gets the headlines. Consolidation is where the savings come from. Karpenter continuously re-evaluates whether the current fleet could be replaced by a cheaper fleet that still satisfies every running pod. It considers three moves, in order of preference:

• Deletion. Can this node's pods all fit on other existing nodes? If yes, cordon it, drain it, terminate it. Pure win, no replacement cost.
• Replacement. Can this node be replaced by a single cheaper node that still fits everything? A m5.4xlarge running workloads that only need 8 vCPU becomes a m5.2xlarge. Karpenter launches the replacement first, waits for it to be ready, then drains the original.
• Multi-node consolidation. Can several underutilized nodes be replaced by a smaller number of nodes? This is the expensive computation and Karpenter rate-limits how aggressively it attempts it.

Teams migrating from Cluster Autoscaler with consolidation enabled commonly see node-hour costs fall 20 to 40 percent. Almost none of that comes from smarter provisioning. It comes from deleting the padding that manual node group sizing always requires.

Part 4: Reading a NodePool Properly

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    metadata:
      labels:
        team: platform
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
      disruption:
        consolidationPolicy: WhenEmptyOrUnderutilized
        consolidateAfter: 1m
        budgets:
          - nodes: "10%"
          - nodes: "0"
          schedule: "0 9 * * mon-fri"
          duration: 8h
      limits:
        cpu: "1000"
        memory: 4000Gi
      weight: 10

Line by line, what each piece is buying you:

• requirements are constraints, not selections. You are describing the space of acceptable instances and letting Karpenter pick from inside it.
• instance-generation: Gt 5 is quietly one of the highest-value lines here: it excludes older generations that are often more expensive per unit of performance, without you naming a single instance type.
• Including both amd64 and arm64 lets Graviton into the candidate pool. If your images are multi-arch, this is frequently a 15 to 20 percent saving for zero configuration effort. If they are not multi-arch, remove arm64 or you will get nodes your pods cannot run on.
• expireAfter: 720h forces every node to be replaced after 30 days. This is your patching story. Nodes get recycled onto fresh AMIs continuously instead of accumulating drift until someone runs a scary cluster-wide upgrade.
• disruption.budgets is the part production teams should not skip. The first entry caps voluntary disruption at 10 percent of nodes at a time. The second pins it to zero during weekday business hours. Karpenter will still handle involuntary events like Spot interruptions, but it will not choose to churn your fleet during your peak traffic window.
• limits is your blast radius. Without it, a runaway Deployment with a bad replica count can provision a genuinely alarming amount of compute before anyone notices. Set it.
• weight matters once you have more than one NodePool. Higher weight is evaluated first. A common pattern is a high-weight Spot-only pool for stateless work and a lower-weight On-Demand pool as fallback.

Part 5: Where Teams Get Burned

Spot interruption handling is not application resilience. Karpenter subscribes to the EC2 interruption notice via SQS and starts draining within the two-minute window. That is genuinely good. But if your pod takes ninety seconds to warm a cache before it serves useful traffic, you have an application design problem that no scheduler fixes. Test this. Actually terminate a Spot node in staging and watch what your p99 does.

Requests, not limits, drive everything. Karpenter bin-packs against resources.requests. If your teams set requests at 100m CPU "to be safe" while the app actually uses 1.5 cores, Karpenter will pack twelve of them onto a node and you will get CPU throttling that looks like a network problem for three days. Karpenter makes the cost of bad resource requests visible in a way Cluster Autoscaler's padding used to hide.

Instance type sprawl hurts observability. Wide-open requirements mean your cost dashboards fill with dozens of instance types and your capacity trends get noisy. Narrowing instance-category and instance-generation trades a small amount of theoretical savings for a lot of operational clarity. Most mature setups do this deliberately.

Underutilized does not mean empty. WhenEmptyOrUnderutilized will move running pods. If you have workloads that genuinely hate rescheduling, either use WhenEmpty, raise consolidAfter, or define proper PodDisruptionBudgets. Karpenter respects PDBs, but only ones that exist.

Do not run Karpenter on nodes Karpenter manages. The controller needs somewhere stable to live: a small managed node group, or Fargate. If Karpenter consolidates the node it is running on, you get a genuinely annoying failure mode.

The Bigger Shift

Karpenter is a signal of where infrastructure is heading generally: away from static pre-provisioned capacity, toward just-in-time capacity shaped by real demand signals. The same philosophy shows up in Fargate, in Lambda's scale-to-zero model, and in how AWS increasingly prices compute. If you are still maintaining a spreadsheet of node groups by hand, the fix is not a better spreadsheet. It is letting the scheduler tell the infrastructure layer what it actually needs, and trusting the infrastructure layer to answer honestly. That is the real lesson underneath the tool, and it outlives the tool.

If you have migrated a cluster from Cluster Autoscaler to Karpenter, I would like to hear what surprised you. The gap between the docs and the first month in production is where the interesting material lives.