If kubelet Is a Worker Node Component, How Does It Manage Static Pods on the Master Node?

This is one of the most disorienting things you can experience in Kubernetes. You delete something. It comes back. You didn't write a Deployment. There is no ReplicaSet. The pod just refuses to leave.

The answer has nothing to do with Deployments or ReplicaSets. It has everything to do with a concept that most tutorials either explain wrong or skip entirely — and that gap follows engineers from their first cluster all the way into senior-level interviews.

The Interview Question

Three months after that exact situation, you're in an interview for a senior Kubernetes engineer role. The interviewer puts down their pen and asks:

Most people pause here. Not because the answer is technically complicated. But because the question contains a wrong assumption they absorbed from a diagram and never questioned.

Fix the assumption and the answer is immediate. Keep the assumption and you spend two minutes constructing an elaborate non-answer while the interviewer waits.

Let's fix it.

What Every Tutorial Gets Wrong

Here's the diagram you've seen in 90% of Kubernetes intro courses:

  What every tutorial shows:

  Master Node                    Worker Node
  ├── API Server                 ├── kubelet   ← "kubelet lives here"
  ├── Scheduler                  ├── kube-proxy
  ├── Controller Manager         └── Application Pods
  └── etcd

  Logical conclusion: kubelet = worker node thing.
  Understandable. Also wrong.

Makes sense. Master node does the thinking. Worker nodes do the running. kubelet manages pods on workers. Clean separation. Logical model.

Most people build on top of that diagram and produce an answer like this:

The problem: this diagram shows roles, not what is actually installed on each node. It is a simplified teaching model. And one piece missing from that diagram is responsible for every mistake this article covers.

The Mental Model — Read This Before Anything Else

Before we go into any technical depth, here is the analogy that makes everything click. Think of how food delivery works.

Hold that model. Everything below maps to the same pattern — except with YAML files and systemd services instead of fridge notes and back doors.

What kubelet Actually Is

Here is the one sentence that fixes the wrong assumption:

kubelet is a node agent. It runs on every node in a Kubernetes cluster — including the control-plane node.

It is not a worker-node component. It never was. The tutorials that drew it only on the worker side were simplifying. That simplification has been quietly causing confusion at every level — beginners, mid-level engineers, and seniors who have been using Kubernetes for years without ever questioning the diagram.

The corrected picture:

  What actually runs on each node:

  Control Plane Node             Worker Node
  ├── kube-apiserver             ├── kubelet
  ├── kube-scheduler             ├── kube-proxy
  ├── kube-controller-manager    └── Application Pods
  ├── etcd
  ├── kubelet      ← yes, here too
  └── kube-proxy

The difference between a control-plane node and a worker node is not about which software is installed. It is about which control-plane services are present and what workloads are allowed to run there. kubelet is on both types of node. Always has been.

Q1: Does kubelet Only Run on Worker Nodes?

The wrong answer: yes, kubelet is a worker-node component.

This is like saying building managers only work at branch offices, not at headquarters. Every building has one — doesn't matter how important the building is. The fact that the HQ has more critical things happening inside doesn't mean nobody manages the building.

kubelet is the building manager. It runs on every node. On worker nodes it manages application pods. On the control-plane node it manages the control-plane pods — API Server, etcd, Scheduler, Controller Manager. Same job. Different tenants.

Q2: What Are Static Pods?

The wrong answer: “pods that don't move between nodes.”

That is a DaemonSet — a scheduling policy that pins one pod per node. A Static Pod has nothing to do with movement. The word “static” refers to how the pod is defined: by a static file on disk, not through a dynamic API call. That is the entire distinction.

  Normal Pod path:

  kubectl apply -f pod.yaml
       ↓
  API Server (validates, stores in etcd)
       ↓
  Scheduler (picks a node)
       ↓
  kubelet on that node (starts the pod)

  ─────────────────────────────────────

  Static Pod path:

  cp nginx.yaml /etc/kubernetes/manifests/
       ↓
  kubelet (detects file, starts the pod)
       ↓
  Done. Nothing else involved.

A Static Pod is any pod defined by a YAML manifest file placed in a directory that kubelet watches. On most clusters, that directory is:

/etc/kubernetes/manifests/

Drop a file in — pod starts. Remove the file — pod stops. No kubectl apply. No API Server involved. No Scheduler assignment. kubelet handles the full lifecycle on its own.

  You drop a YAML file here:
  /etc/kubernetes/manifests/nginx-static.yaml

  kubelet is watching that directory.
      ↓
  kubelet detects the new file
      ↓
  Pod starts on this node.
      ↓
  No kubectl apply. No API Server call. No Scheduler.
      ↓
  kubectl get pods shows it — as a read-only mirror.

Q3: Who Actually Starts kube-apiserver?

The wrong answer: “the control plane” or “Kubernetes itself.”

Kubernetes cannot start itself any more than a restaurant can cook its own first customer. Someone has to turn the lights on before any of the normal operations begin.

Go look at what lives in the manifests directory on a real cluster:

ls /etc/kubernetes/manifests/

etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

Those four files start the entire control plane. Not a Helm chart. Not a Deployment. Not some hidden bootstrap service. Four YAML files in a directory that kubelet is watching.

When the node boots, kubelet starts as a systemd service — it needs nothing else to run. It reads those four files. It starts all four pods directly, as static pods. The cluster comes alive from those four files and nothing else.

kube-apiserver starts because kube-apiserver.yaml is in that directory. That is the complete answer.

Q4: The Bootstrap Problem

This is where it really clicks — or where most explanations quietly skip the hard bit.

In Kubernetes, creating a pod normally requires the API Server. The Scheduler assigns it to a node. kubelet starts it. That is the chain.

But the API Server is itself a pod. So is etcd. So is the Scheduler.

So: if pods need the API Server to be created — but the API Server is a pod that needs to be created — who goes first? You cannot use the thing to create the thing. The cluster would never boot. This is called the bootstrap problem. It is a genuine circular dependency with no obvious solution.

Static Pods are the solution.

  Node boots up
      ↓
  kubelet starts  (systemd service — needs nothing else to run)
      ↓
  kubelet reads /etc/kubernetes/manifests/
      ↓
  etcd.yaml                     → etcd pod starts
  kube-apiserver.yaml           → API Server starts
  kube-scheduler.yaml           → Scheduler starts
  kube-controller-manager.yaml  → Controller Manager starts
      ↓
  Cluster is alive. Four pods. Four files. That's the whole foundation.

kubelet is a systemd service. It starts on node boot. It does not need the API Server — it reads manifest files from disk. Those files tell it to start etcd, kube-apiserver, kube-scheduler, and kube-controller-manager. kubelet starts all four as static pods. No API Server involved in any of that.

Once those four pods are running, the API Server is alive and the cluster handles everything else through normal channels. But the first boot is kubelet, working completely alone, reading four files.

Q5: Why Does kubectl delete Not Work on Static Pods?

Back to the pod that wouldn't die from the opening.

You run kubectl delete pod nginx-static-control-plane. The pod disappears. Three seconds later it is back.

This is the equivalent of crossing someone's name off the building directory and wondering why they still live in the apartment. The directory is decorative. The person is still there.

Here is what is actually happening. What you see in kubectl get podsfor a static pod is called a mirror pod — a read-only reflection that kubelet registers with the API Server so you can observe the pod using standard tools. But the API Server does not control it. kubelet does.

When you delete the mirror pod, you have removed the reflection. kubelet immediately re-registers it because the manifest file is still on disk and the pod is still running. Nothing changed on kubelet's side. The mirror reappears within seconds.

The only way to actually stop a static pod is to remove the manifest file:

sudo rm /etc/kubernetes/manifests/nginx-static.yaml

File gone → kubelet stops managing it → pod stops → mirror disappears and stays gone.

Q6: What Happens If a Static Pod Crashes at 3 AM?

kubelet does not just start static pods. It keeps watching them.

If the container inside a static pod crashes, kubelet detects it and restarts it. Automatically. No alert needed. No human needed. No other Kubernetes component needed.

  Static pod container crashes at 3 AM
       ↓
  kubelet detects the failure  (watching continuously)
       ↓
  kubelet restarts the container  (no human, no other component needed)
       ↓
  Pod is running again
       ↓
  You wake up. Everything is fine. You never know it happened.

This matters enormously for the control plane. If kube-apiserver crashes at 3 AM — bad memory pressure, a node hiccup, a kernel OOM event — kubelet on the control-plane node catches it and brings it back on its own. By the time anyone opens a laptop, the cluster has already recovered.

Create Your First Static Pod

The best way to understand this is to feel it directly. On a kubeadm cluster or any lab environment, these steps work exactly as described.

Step 1 — Create the manifest file

sudo nano /etc/kubernetes/manifests/nginx-static.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-static
spec:
  containers:
    - name: nginx
      image: nginx:latest

Save the file. Do not run kubectl apply. Do not restart anything. Just save the file and wait five seconds.

Step 2 — Verify it started

kubectl get pods
NAME                          READY   STATUS    NODE
nginx-static-control-plane    1/1     Running   control-plane

The node name is appended automatically: nginx-static-control-plane. That is kubelet's naming convention. Any pod whose name ends with the node name is a static pod managed directly by kubelet from a manifest file on disk.

Step 3 — Try to delete it the wrong way

kubectl delete pod nginx-static-control-plane

Wait a few seconds. Run kubectl get pods again. It is back. Now you have felt it directly instead of just reading about it.

Step 4 — Delete it the right way

sudo rm /etc/kubernetes/manifests/nginx-static.yaml

Wait a few seconds. Pod is gone. Stays gone.

Two Production Disasters

Disaster 1: The Deletion Loop (30 people, startup, staging)

A junior DevOps engineer doing routine kube-system cleanup. They run kubectl get pods -n kube-system and spot pods with names like kube-scheduler-control-plane. Never seen these before. Looks like something stuck from a previous operation.

They delete the pod. It comes back. They delete it six more times. They add --force --grace-period=0. Still comes back. They open a Slack thread: “I can't delete this pod — is the cluster broken?”

Senior engineer joins 20 minutes later. First question:“SSH into the control-plane node and run ls /etc/kubernetes/manifests/.”

There it is. kube-scheduler.yaml. kubelet had been doing its job the entire time. The pod was not stuck — it was the Scheduler, running as a static pod. Had they successfully deleted it, new pods would have stopped being scheduled across the cluster. The deletion loop was the safe outcome.

Time lost: 25 minutes. Root cause: one missing piece of context about static pods.

Disaster 2: The Manifest Edit That Took Down the API Server

A platform team doing a planned upgrade. They need to add a new audit policy flag to kube-apiserver. Standard procedure on this team: edit the manifest in place, let kubelet pick up the change.

An engineer SSHes into the control-plane node and opens /etc/kubernetes/manifests/kube-apiserver.yaml in vim. Types the new flag. Makes a one-character typo in the flag name. Saves the file.

kubelet detects the file change in under two seconds. Restarts kube-apiserver with the broken config. kube-apiserver fails to start.

kubectl stops responding. The team cannot use kubectl to diagnose the problem because there is no API Server to talk to. They have to SSH back to the control-plane node, open the manifest file directly, find the typo, fix it, save. kubelet picks up the corrected file and restarts kube-apiserver. Cluster back online. Total downtime: 8 minutes.

The Wall of Shame

Six mistakes. All extremely common. Most made by experienced engineers.

Best Practices

1. Never touch manifest files without direct SSH access to the control-plane node. If the API Server goes down mid-edit, kubectl goes with it. You will need SSH to fix it anyway — open it before you start.
2. Validate YAML before saving to the manifests directory. Use kubectl --dry-run=client -f file.yaml before moving it in, or run yamllint on it. kubelet applies immediately with no confirmation dialog.
3. Recognize static pods by their name suffix. Pod name ends with the node name (-control-plane, -node01, etc.) = static pod. Manage it through the file, not through kubectl.
4. Use journalctl -u kubelet -f when a control-plane component won't start. The manifest might be syntactically valid but kubelet might be rejecting a specific flag. kubelet logs will tell you exactly why.
5. Do not use static pods for application workloads. Static pods on the control-plane are for control-plane components only. For everything else — Deployments, DaemonSets, StatefulSets.
6. Confirm the manifests directory path if something seems off. The default is /etc/kubernetes/manifests/ but it is configurable in the kubelet config. Run cat /var/lib/kubelet/config.yaml | grep staticPodPath to confirm what your cluster is actually watching.

FAQ

People Also Ask

Interview Corner

🎤 The 60-Second Answer

The pod that wouldn't die at 2:17 AM was not broken. It was doing exactly what it was designed to do. kubelet was watching the manifest file on disk, saw that the pod should be running, and put it back every time you deleted the mirror. The cluster was working correctly. You just didn't have the model yet.

The interview question — “if kubelet is a worker node component, how does it manage static pods on the master?” — is not really about static pods. It is about whether you understood what kubelet actually is. Fix that one sentence and everything else follows: the static pods, the bootstrap problem, the deletion behavior, the mirror pod, the self-healing.

One wrong sentence in a tutorial caused all of it. One correct sentence clears all of it.