If Karpenter Runs on One Node, How Does It Know When Other Nodes Need More Capacity?

The NodePool had been locked to two instance types three weeks earlier — “to control costs.” Both were sold out across every availability zone. It was Black Friday. AWS was not holding them in reserve.

Karpenter was working perfectly. The configuration was the failure. And the reason it took 45 minutes to diagnose — instead of 4 minutes — is that most engineers don't fully understand what Karpenter is, what it watches, and what it controls. This article fixes that.

The Interview Question

Six months after that incident, you're interviewing for a Senior Platform Engineer role. The interviewer is taking notes, then stops and asks:

Most people answer this by describing what they wish Karpenter did: monitoring node metrics, watching CPU and memory, responding to thresholds. That answer is completely wrong. And the interviewer knows it.

The correct answer requires understanding one architectural decision that Karpenter makes — and that decision changes everything about how you configure, debug, and trust it in production.

The Simple Answer (The Trap)

Here is the naive mental model of how Karpenter works — the one most people carry into interviews:

Let's answer all five. But first — the mental model that makes everything click.

The Mental Model — Read This Before Anything Else

Before any technical depth, here is the analogy. Think of how a staffing agency works for a company with a permanent team and occasional surge demand.

Hold that model. Karpenter is the staffing agency. The API Server is the HR system that tells it there is an open role. EC2 is the talent pool. The new hire (node) is real — but the agency (Terraform) that built the company never wrote them into its books.

What Karpenter Replaces

Before Karpenter, EKS scaling meant Cluster Autoscaler. Cluster Autoscaler has one job: adjust the node count inside pre-existing Node Groups.

  Traditional EKS scaling:

  EKS Cluster
  │
  ├── Managed Node Group A  (max: 5)
  │   ├── Node 1
  │   ├── Node 2
  │   └── Node 3    ← Cluster Autoscaler adds nodes here
  │
  └── Managed Node Group B  (max: 3)
      ├── Node 4
      └── Node 5    ← Or here. Only inside existing groups.

  Cluster Autoscaler can scale within groups.
  It cannot create new groups. It cannot pick instance types.
  It scales the count. Karpenter picks the instance.

Cluster Autoscaler cannot pick instance types. It cannot create new node groups. It cannot respond to topology constraints or GPU requirements automatically. When your node group hits max capacity and you need a different instance type, Cluster Autoscaler waits. You configure. You wait more.

Karpenter removes the node group middleman entirely.

Q1: Does Karpenter Create New Node Groups?

The wrong answer: yes, Karpenter creates and manages Managed Node Groups dynamically.

This is like saying a hotel concierge builds a new hotel wing every time a guest needs a room. The concierge calls room service. The wing was never part of the plan.

Karpenter calls the AWS EC2 Fleet API directly and launches individual instances. No Managed Node Group is created. No CloudFormation stack. No Auto Scaling Group. The instance bootstraps using the EKS node bootstrap script, registers itself with the cluster, and gets tagged with karpenter.sh/nodepool. That tag is how Karpenter tracks ownership. The node is real. It just has no group.

  What actually happens:

  New pod arrives — no node has capacity
       │
       ▼
  Kubernetes Scheduler: "I cannot place this pod"
       │
       ▼
  Pod status → Pending (condition: Unschedulable)
       │
       ▼
  Karpenter watches API Server — sees the Pending pod
       │
       ▼
  Karpenter calculates: what instance satisfies this pod's
  requests + limits + nodeSelector + tolerations + affinity?
       │
       ▼
  Karpenter calls AWS EC2 Fleet API directly
       │
       ▼
  EC2 instance launches, bootstraps, joins EKS cluster
       │
       ▼
  Kubernetes Scheduler places the pod on the new node
       │
       ▼
  Pod is Running. No Node Group was created.

Q2: Why Is Karpenter a Deployment, Not a DaemonSet?

The wrong answer: “because it needs to monitor resource usage on every node.”

This is the staffing agency equivalent of requiring the recruiter to sit at every employee's desk simultaneously to see if anyone needs help. The recruiter needs one desk and a phone. They respond to requests — they don't surveil the building.

Karpenter is a Kubernetes controller. Controllers watch the API Server for events and reconcile state. They do not run on every node — they run once (or twice for HA) and react. One replica is enough because the API Server is the single source of truth for everything Karpenter needs to know.

DaemonSets are for node-local agents: things that need access to a specific node's filesystem, kernel, or network interfaces. Examples:

Q3: How Does Karpenter Know When to Provision a New Node?

The wrong answer: “it monitors CPU and memory thresholds on existing nodes.”

Karpenter has never looked at a CPU metric. That is the HPA's job. Karpenter does not care how loaded your nodes are. It cares about exactly one thing: pods that the Kubernetes Scheduler declared it could not place.

Here is the sequence that actually happens:

  Karpenter lives on ONE node.
  It watches ONE thing: the Kubernetes API Server.

  ┌─────────────────────────────────────────────────────┐
  │                 Kubernetes API Server               │
  │   (stores pod state, node state, all events)        │
  └─────────────────────────┬───────────────────────────┘
                            │  watches for Pending pods
                            ▼
                     ┌──────────────┐
                     │  Karpenter   │  ← runs as a Deployment
                     │  (1-2 pods)  │    on managed node group
                     └──────┬───────┘
                            │  calls
                            ▼
                     AWS EC2 Fleet API
                            │
                            ▼
                     New EC2 instance
                            │
                     joins EKS cluster

1. A new pod arrives with resource requests the current nodes cannot satisfy.
2. The Kubernetes Scheduler tries every node. None fits. It marks the pod Pending with condition Unschedulable.
3. Karpenter is watching the API Server for exactly this event.
4. Karpenter reads the pod spec: CPU requests, memory requests, nodeSelector, tolerations, topology spread constraints, affinity rules.
5. Karpenter calculates the cheapest instance type from its NodePool that satisfies all of those constraints.
6. Karpenter calls the AWS EC2 Fleet API. The instance launches.
7. The new node registers with the cluster. The Scheduler places the pending pod on it.

Karpenter does not care which node the pod was trying to run on. It does not care how loaded your existing nodes are. It cares only that a pod is waiting and no current node will take it.

Q4: Will Terraform Delete Karpenter Nodes?

The wrong answer: yes, terraform apply or terraform destroy will remove them.

Terraform can only destroy what it created. Karpenter nodes were born outside Terraform's state file — like contractors hired by a staffing agency that bypassed the company's HR system. Terraform's destroy command has no idea they exist.

  Terraform manages:              Karpenter manages:
  ─────────────────               ─────────────────
  VPC                             Worker nodes (EC2 instances)
  EKS Cluster                     NodePool CRDs
  IAM Roles/Policies              EC2NodeClass CRDs
  Baseline Node Group             Node lifecycle (launch, drain, terminate)
  Karpenter Helm install

  terraform.tfstate knows about   terraform.tfstate does NOT know about
  everything on the left.         anything on the right.

When you run terraform destroy, Terraform reads its state file. It sees the VPC, the EKS cluster, the IAM roles, the managed node group. It does not see the 20 EC2 instances Karpenter launched last night. It deletes everything it knows about — and leaves the Karpenter nodes as orphaned EC2 instances with no cluster to connect to.

Those orphaned instances keep running and billing you until you terminate them manually or until Karpenter terminates them (which it cannot, because the cluster it talks to is gone).

Q5: What Is the Difference Between NodePool and EC2NodeClass?

The wrong answer: “they're just the new names for Node Groups.”

NodePool says what you want. EC2NodeClass says how to build it. Confusing them is like confusing a job description with an employment contract. One defines the requirements. The other defines the terms.

NodePool — the scheduling constraints

NodePool defines what Karpenter is allowed to provision: which instance families, which architectures, which availability zones, which capacity types (Spot vs On-Demand), what taints to apply, and — critically — a resource budget cap that prevents runaway scaling.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["m5.large","m5.xlarge","m5.2xlarge",
                   "m6i.large","m6i.xlarge","m6i.2xlarge",
                   "c5.xlarge","c5.2xlarge"]
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s
  limits:
    cpu: "200"
    memory: 400Gi

EC2NodeClass — the AWS-specific config

EC2NodeClass defines the AWS details that every instance launched by this NodePool will use: which AMI family, which subnets (via tags), which security groups, which IAM instance profile.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2023
  role: "KarpenterNodeRole-my-cluster"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster"
  amiSelectorTerms:
    - alias: al2023@latest

One NodePool references one EC2NodeClass. You can have multiple NodePools pointing at the same EC2NodeClass — for example, a GPU NodePool and a CPU NodePool both using the same subnet and security group config, but with different instance type constraints.

Q6: What Happens If the Node Running Karpenter Is Terminated?

Most people answer: “the whole cluster breaks.” The honest answer is: it depends entirely on one configuration decision.

Karpenter is a Deployment. If its node goes down, Kubernetes reschedules it on another available node. Existing workloads keep running. New pending pods wait — but only for the time it takes Karpenter to restart. That is usually under 60 seconds.

But there is a scenario where this goes badly wrong.

  The Karpenter bootstrap problem:

  Who manages Karpenter?    →  A managed node group (Terraform)
  Who manages app nodes?    →  Karpenter
  Who manages Karpenter's node?  →  NOT Karpenter

  ┌──────────────────────────────────────────────────────┐
  │  Managed Node Group (Terraform)                      │
  │  ├── karpenter-xxxx  ← lives here, safe from         │
  │  └── coredns-xxxx      Karpenter consolidation       │
  └──────────────────────────────────────────────────────┘

  ┌──────────────────────────────────────────────────────┐
  │  Karpenter-provisioned nodes                         │
  │  ├── app-pod-a                                       │
  │  └── app-pod-b                                       │
  └──────────────────────────────────────────────────────┘

  If Karpenter runs on a node it manages → it can
  consolidate its own floor → evict itself → deadlock.

If Karpenter manages all nodes in the cluster — no baseline managed node group — and consolidation is enabled, Karpenter can look at the node it runs on, decide it is underutilized, and begin draining it. Karpenter gets evicted. Its pod becomes Pending. No node is available for it to reschedule onto. No Karpenter means no provisioner. The cluster cannot scale. New pods pile up as Pending indefinitely.

This is the Karpenter bootstrap problem — directly analogous to the static pod bootstrap problem. The provisioner cannot provision for itself.

Two Production Disasters

Disaster 1: The NodePool That Wouldn't Scale on Black Friday

An e-commerce company, 50-node EKS cluster, Black Friday traffic. Three weeks earlier, a platform engineer had locked the NodePool to m5.large and m5.xlarge only — “for cost predictability.”

At 2:17 AM under 4x traffic load, both instance types were sold out across all three availability zones. AWS EC2 Fleet API returned InsufficientInstanceCapacityfor every provisioning attempt. Karpenter retried, got the same error, and stopped. 47 pods stayed Pending. The site partially degraded for 45 minutes.

The fix: added six additional instance types to the NodePool requirements — m6i.large, m6i.xlarge, c5.xlarge, c5.2xlarge, r5.large, r5.xlarge. Pods scheduled within 90 seconds of the change.

The restriction was written to save money. It cost an estimated $200k in lost revenue over 45 minutes. Karpenter's entire value is picking the right instance automatically. Locking it to one type defeats the purpose entirely.

Disaster 2: Karpenter Consolidated Its Own Floor

A platform team enabling Karpenter node consolidation for the first time on a production cluster. They set consolidationPolicy: WhenUnderutilized with consolidateAfter: 30s. The Karpenter Deployment had no node selector — it could run on any node, including Karpenter-managed nodes.

At 3:10 AM, traffic dropped to night-time levels. Karpenter scanned the cluster. The node running Karpenter was underutilized. Karpenter began draining it. The Karpenter pod was evicted. Its pod became Pending. No provisioner to schedule it onto a new node. No new nodes were provisioned. Karpenter stayed Pending for 6 hours until the on-call engineer came online at 9 AM.

The Wall of Shame

Six mistakes. All extremely common. Several made by senior engineers who knew better.

Best Practices

1. Always maintain a baseline managed node group for system components. Karpenter, CoreDNS, aws-node, kube-proxy — pin them to managed nodes with a nodeSelector and taint. Never let Karpenter consolidate its own floor.
2. Set resource limits on every NodePool before production. spec.limits.cpu and spec.limits.memory are not optional. They are the only thing preventing runaway provisioning during a load test or HPA misconfiguration.
3. Allow 8–10 instance types, not 1–2. Use instance families (m5.*, m6i.*, c5.*) and let Karpenter optimize within them. AWS instance availability is not guaranteed. Diversity is your hedge.
4. Set PodDisruptionBudgets before enabling consolidation. Every stateful workload, every single-replica Deployment, and every system component should have a PDB before you turn on WhenUnderutilized.
5. Disable Cluster Autoscaler completely when migrating to Karpenter. They do not coexist gracefully. Pick one. For new clusters, Karpenter. For existing clusters, migrate then disable CAS in a single planned window.
6. Delete Karpenter-managed nodes before terraform destroy. kubectl delete nodes -l karpenter.sh/nodepool — run this first, wait for termination, then run your Terraform commands.
7. Use karpenter.sh/do-not-disrupt: "true" for critical pods. This annotation tells Karpenter never to evict this pod during consolidation. Use it on anything that cannot tolerate voluntary disruption.

FAQ

People Also Ask

Interview Corner

🎤 The 60-Second Answer

The 47 pending pods at 2:17 AM were not a Karpenter failure. Karpenter was doing exactly what it was configured to do: provision instances from the allowed list. The allowed list had two items. Both items were sold out. The cluster was working correctly. The configuration was the failure.

The interview question — “how does Karpenter know when pods on other nodes need more capacity?” — is not really about Karpenter's architecture. It is about whether you understand the API-Server-centric model of Kubernetes. Everything in Kubernetes flows through the API Server. Karpenter is just the part that listens for the signal that the Scheduler already broadcasts.

Understand that, and every Karpenter configuration decision — NodePool constraints, resource limits, Spot vs On-Demand, consolidation policy — becomes obvious reasoning, not memorized settings.