Kubernetes Troubleshooting Guide: A Production Debugging Runbook for Every Failure Mode

Kubernetes failures are not random. Every failure mode has a fingerprint — a specific combination of pod status, exit code, event message, and resource state that points to one root cause. This runbook gives you the exact commands and decision logic to read those fingerprints under pressure.

This is not a theoretical guide. Every section is structured as a live debugging session: what you see in the terminal, what command to run next, and what to do when you find the answer.

1. The Troubleshooting Mental Model: Top-Down Approach

The single biggest mistake engineers make during an incident is jumping to logs before understanding the scope of the failure. Start at the top of the hierarchy and work down. If 10 pods are failing, the root cause is almost certainly at the namespace, node, or infrastructure layer — not inside any individual container.

The Hierarchy

1. Namespace — Quota, LimitRange, RBAC, NetworkPolicy scope
2. Workload — Deployment, StatefulSet, DaemonSet desired vs available replicas
3. Pod — Status, restart count, node assignment
4. Container — Exit code, logs, resource usage
5. Node — Conditions, pressure, kubelet health
6. Network — CNI, Service endpoints, DNS, NetworkPolicy
7. Storage — PVC binding, StorageClass, disk pressure

Top-Down Decision Tree

  INCIDENT REPORTED
        │
        ▼
  ┌─────────────────────────────────────────┐
  │  kubectl get ns                          │
  │  Namespace exists? Quota/LimitRange?     │
  └──────────────────┬──────────────────────┘
                     │
                     ▼
  ┌─────────────────────────────────────────┐
  │  kubectl get deploy,sts,ds -n <NS>       │
  │  AVAILABLE replicas == DESIRED?          │
  └──────────────────┬──────────────────────┘
                     │
                     ▼
  ┌─────────────────────────────────────────┐
  │  kubectl get pods -n <NS> -o wide        │
  │  All Running?  Check STATUS column       │
  └──────────────┬──────────────────────────┘
       ┌─────────┼──────────────────────────┐
       │         │                          │
  Pending   CrashLoop              ImagePullBackOff
       │    OOMKilled              ContainerCreating
       │    RunContainerError      Init:X/N
       │         │                          │
       ▼         ▼                          ▼
  Check:     kubectl logs              Check:
  Resources  --previous                Registry auth
  Taints     kubectl describe          Image tag
  PVC        (Events section)          Network policy
  Quota      Exit code                 Secret exists
       │         │                          │
       └─────────┴──────────────────────────┘
                     │
                     ▼
  ┌─────────────────────────────────────────┐
  │  Pod Running but no traffic?             │
  │  kubectl get ep <SVC> -n <NS>           │
  │  Endpoints populated?                    │
  └──────────────────┬──────────────────────┘
                     │
           ┌─────────┴──────────┐
        Empty EP             Endpoints exist
           │                    │
    readinessProbe         Check Ingress
    failing                NetworkPolicy
    port mismatch          DNS
           │                    │
           └─────────┬──────────┘
                     │
                     ▼
  ┌─────────────────────────────────────────┐
  │  Node-level issues?                      │
  │  kubectl get nodes                       │
  │  kubectl describe node <NODE>            │
  │  Conditions: MemoryPressure DiskPressure │
  └─────────────────────────────────────────┘

Pod Failure Mode Flowchart

  Pod created
      │
      ▼
  Scheduler assigns node?
  ├── NO  ──▶  STATUS: Pending
  │            Check: resources, taints, affinity, PVC, quota
  │
  └── YES ──▶  kubelet pulls image?
               ├── NO  ──▶  STATUS: ImagePullBackOff / ErrImagePull
               │            Check: registry auth, image name/tag, network
               │
               └── YES ──▶  Init containers pass?
                            ├── NO  ──▶  STATUS: Init:X/N
                            │            Check: init container logs
                            │
                            └── YES ──▶  Container starts?
                                         ├── NO  ──▶  STATUS: RunContainerError
                                         │            Check: command, permissions
                                         │
                                         └── YES ──▶  Container stays running?
                                                      ├── NO  ──▶  CrashLoopBackOff
                                                      │            exit 137: OOMKilled
                                                      │            exit 1:   app crash
                                                      │            exit 127: cmd not found
                                                      │
                                                      └── YES ──▶  Readiness probe passes?
                                                                   ├── NO  ──▶  Running but not in Service endpoints
                                                                   └── YES ──▶  HEALTHY ✓

2. Pod Troubleshooting: Every Failure Mode

2.1 Pending Pods

A pod stuck in Pending has been accepted by the API server but the scheduler has not assigned it to a node. The scheduler logs its exact reason in the pod events. There are five common causes.

What You See

kubectl get pods -n payments shows STATUS: Pending with no node assigned in the NODE column.

Commands to Run

# Step 1 — See what the scheduler is saying
kubectl describe pod <POD_NAME> -n <NAMESPACE>
# Look at the Events section at the bottom:
#   "0/3 nodes are available: 3 Insufficient memory"  → resource constraint
#   "0/3 nodes are available: 3 node(s) had taint"    → taint/toleration mismatch
#   "0/3 nodes are available: 3 pod has unbound PVC"  → PVC not bound

# Step 2 — Check namespace quota
kubectl describe resourcequota -n <NAMESPACE>
# Look for: Used vs Hard — if Used == Hard, new pods will be rejected

# Step 3 — Check LimitRange (often missed!)
kubectl get limitrange -n <NAMESPACE>
kubectl describe limitrange -n <NAMESPACE>
# If LimitRange requires requests/limits and pod has none → rejected immediately

# Step 4 — Check node capacity vs requests
kubectl describe nodes | grep -A 5 "Allocated resources"
# Shows cpu/memory requested vs allocatable per node

# Step 5 — Check taints on nodes
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Step 6 — Check PVC binding status
kubectl get pvc -n <NAMESPACE>
# STATUS: Pending → no matching PV or StorageClass provisioner issue

Root Causes and Fixes

• Insufficient resources: Event says "0/3 nodes are available: 3 Insufficient memory." Fix: reduce resource requests, add nodes, or scale the cluster.
• Taint/toleration mismatch: Event says "node(s) had taint that the pod didn't tolerate." Fix: add tolerations to the pod spec, or remove the taint from the node.
• Node affinity/anti-affinity: Event says "node(s) didn't match Pod's node affinity/selector." Fix: verify node labels match the pod's nodeSelector or nodeAffinity rules.
• PVC not bound: Pod requires a PVC that is still Pending. Check StorageClass, PV availability, and provisioner logs.
• ResourceQuota / LimitRange: Namespace quota exhausted, or LimitRange requires resource requests the pod doesn't specify. This was the Day Two incident described above. Fix: add resource requests to the pod, or increase quota.

2.2 CrashLoopBackOff

CrashLoopBackOff means the container is starting, crashing, and Kubernetes is exponentially backing off before restarting it. The back-off starts at 10 seconds and doubles up to 5 minutes. The key data is in the exit code and the previous container logs.

Commands to Run

# Step 1 — Get exit code and restart count
kubectl get pod <POD_NAME> -n <NAMESPACE>
# RESTARTS column: high number = CrashLoopBackOff

# Step 2 — Fetch logs from the PREVIOUS (crashed) container
kubectl logs <POD_NAME> -n <NAMESPACE> --previous
kubectl logs <POD_NAME> -n <NAMESPACE> --previous -c <CONTAINER_NAME>

# Step 3 — Check Events and last state
kubectl describe pod <POD_NAME> -n <NAMESPACE>
# Look for:
#   Last State: Terminated  Reason: OOMKilled  Exit Code: 137
#   Last State: Terminated  Reason: Error      Exit Code: 1
#   Last State: Terminated  Reason: Error      Exit Code: 127  (command not found)

# Exit code cheat sheet:
#   0   → clean exit (should not crash)
#   1   → application error (check app logs)
#   2   → bash misuse
#   126 → permission denied on command
#   127 → command not found (wrong entrypoint / missing binary)
#   128 → invalid exit argument
#   137 → OOMKilled (128 + SIGKILL signal 9)
#   143 → graceful SIGTERM (expected during shutdown)

# Step 4 — If exit 127: inspect the image entrypoint
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].command}'
kubectl run debug-shell --image=<YOUR_IMAGE> --restart=Never -it --rm -- /bin/sh

Decision by Exit Code

• Exit 137 (OOMKilled): Container ran out of memory. See Section 2.3.
• Exit 1: Application-level error. The answer is in the logs: kubectl logs --previous. Common causes: missing env vars, failed DB connection, bad config file.
• Exit 127: The command specified in the container spec does not exist in the image. Check the command / args field and verify the binary is in the image at that path.
• Exit 126: Command found but not executable (permissions). The binary may lack execute permission.
• Exit 143: Clean SIGTERM received. If this happens on startup, a preStop hook or a liveness probe may be killing the container before it finishes initializing.

2.3 OOMKilled

OOMKilled means the Linux kernel OOM killer killed the container process because it exceeded its cgroup memory limit. Unlike a node-level OOM, this is deterministic — the container hit exactly its resources.limits.memory value.

Commands to Run

# Step 1 — Confirm OOMKilled
kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A 3 "Last State"
# Last State: Terminated  Reason: OOMKilled

# Step 2 — See which container was killed
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{range .status.containerStatuses[*]}{.name}{": "}{.lastState.terminated.reason}{"
"}{end}'

# Step 3 — Check current memory limits
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{range .spec.containers[*]}{.name}{": requests="}{.resources.requests.memory}{" limits="}{.resources.limits.memory}{"
"}{end}'

# Step 4 — See live memory usage (requires metrics-server)
kubectl top pod <POD_NAME> -n <NAMESPACE> --containers

# Step 5 — Watch memory trend over time
kubectl top pods -n <NAMESPACE> --containers --sort-by=memory

# Step 6 — Check VPA recommendation (if VPA is installed)
kubectl get vpa -n <NAMESPACE>
kubectl describe vpa <VPA_NAME> -n <NAMESPACE>
# Look for: Recommendation → Target memory

Fix Strategy

1. Determine if the OOM is expected (insufficient limit) or a memory leak (limit is reasonable but memory grows unboundedly).
2. If insufficient limit: increase resources.limits.memory. Set the limit to at least 150% of normal peak usage with headroom for GC spikes (JVM, Go, Node.js all have GC pauses that temporarily spike memory).
3. If memory leak: fix the application. Adding memory is a short-term mitigation. Use heap profiling, pprof (Go), jmap/jstack (JVM), or node-inspector (Node.js) to find the leak.
4. Consider Vertical Pod Autoscaler (VPA) to automatically right-size memory limits based on historical usage.

2.4 ImagePullBackOff / ErrImagePull

The kubelet cannot pull the container image. ErrImagePull is the immediate failure; ImagePullBackOff is the exponential backoff state after repeated failures.

Commands to Run

# Step 1 — Describe the pod for the exact error
kubectl describe pod <POD_NAME> -n <NAMESPACE>
# Events section:
#   Failed to pull image "myregistry.io/app:latest": rpc error: ... unauthorized
#   Failed to pull image "myregistry.io/app:v2.5": ... not found
#   Failed to pull image: ... dial tcp: i/o timeout  (network/registry unreachable)

# Step 2 — Verify the image tag actually exists
# From your workstation:
docker pull myregistry.io/app:v2.5
# Or check registry API:
curl -s https://myregistry.io/v2/app/tags/list

# Step 3 — Check imagePullSecret is attached
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.imagePullSecrets}'
kubectl get secret <SECRET_NAME> -n <NAMESPACE>

# Step 4 — Verify secret has valid credentials
kubectl get secret <SECRET_NAME> -n <NAMESPACE> -o jsonpath='{.data..dockerconfigjson}' | base64 -d | jq .

# Step 5 — Test registry reachability from inside the cluster
kubectl run registry-test --image=curlimages/curl --restart=Never -it --rm --   curl -v https://myregistry.io/v2/

# Step 6 — Check NetworkPolicy blocking registry egress
kubectl get networkpolicy -n <NAMESPACE>

Root Cause Checklist

• Unauthorized (401/403): The imagePullSecret is missing, expired, or points to the wrong registry URL. Recreate the secret with fresh credentials.
• Not found (404): The image tag does not exist. Verify in the registry. The tag may have been garbage collected, or there is a typo (latest vs v1.2).
• Network timeout: The node cannot reach the registry. Check Security Group rules, VPC endpoint configs, and whether a NetworkPolicy is blocking egress on port 443.
• Wrong registry: Image name missing registry prefix defaults to Docker Hub. Private images must include the full registry FQDN.

2.5 Init:X/N Stuck

Init:0/2 means the pod has 2 init containers and the first (index 0) is either running or has failed. Init containers run sequentially — if one fails, the pod restarts it (subject to restartPolicy) and the main containers never start.

Commands to Run

# Step 1 — See init container status
kubectl get pod <POD_NAME> -n <NAMESPACE>
# STATUS: Init:0/2  means first init container running, 2 total

# Step 2 — Get logs from the failing init container
kubectl logs <POD_NAME> -n <NAMESPACE> -c <INIT_CONTAINER_NAME>
kubectl logs <POD_NAME> -n <NAMESPACE> -c <INIT_CONTAINER_NAME> --previous

# Step 3 — List init container names
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.initContainers[*].name}'

# Step 4 — Describe for events and init container status
kubectl describe pod <POD_NAME> -n <NAMESPACE>
# Init Containers section shows state of each

Common init container failures: waiting for a database to be ready (check the health check logic), missing secrets or configmaps (check volume mounts), or a migration script that fails (check the init container logs for migration output).

2.6 Terminating Stuck

A pod stuck in Terminating for more than a few minutes has either a finalizer preventing deletion, a kubelet that is not processing the deletion (node issue), or a very long preStop hook.

Commands to Run

# Step 1 — Check how long it has been Terminating
kubectl get pod <POD_NAME> -n <NAMESPACE>
# AGE column shows total age; check when deletion was triggered:
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.deletionTimestamp}'

# Step 2 — Check for finalizers (most common cause)
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.finalizers}'
# Non-empty output = pod has finalizers that must be cleared first

# Step 3 — Remove finalizers to unblock deletion
kubectl patch pod <POD_NAME> -n <NAMESPACE> -p '{"metadata":{"finalizers":[]}}' --type=merge

# Step 4 — Force delete (LAST RESORT — dangerous on stateful workloads)
# WARNING: This can cause split-brain on StatefulSets, data loss on PVCs
kubectl delete pod <POD_NAME> -n <NAMESPACE> --force --grace-period=0

# Step 5 — Check if kubelet on the node is healthy
kubectl get node <NODE_NAME>
# NotReady node = kubelet cannot process deletions

2.7 ContainerCreating Stuck

ContainerCreating means the pod has been scheduled to a node but the kubelet cannot start the container. The most common causes are volume mount failures (PVC, Secret, ConfigMap not found) and CNI failures.

Commands to Run

# Step 1 — Describe pod immediately
kubectl describe pod <POD_NAME> -n <NAMESPACE>
# Common events to look for:
#   "Unable to attach or mount volumes... secret not found"
#   "Unable to attach or mount volumes... configmap not found"
#   "Unable to mount volumes for pod... PVC not bound"
#   "networkPlugin cni failed..."

# Step 2 — Verify Secrets and ConfigMaps exist
kubectl get secret <SECRET_NAME> -n <NAMESPACE>
kubectl get configmap <CM_NAME> -n <NAMESPACE>

# Step 3 — Check PVC status
kubectl get pvc -n <NAMESPACE>
# STATUS must be Bound

# Step 4 — Check CNI health (if CNI errors in events)
kubectl get pods -n kube-system | grep -E "calico|flannel|cilium|weave"
# All CNI pods should be Running

# Step 5 — Check node CNI socket on problem node
kubectl debug node/<NODE_NAME> -it --image=busybox -- nsenter -t 1 -m -u -i -n --   ls /var/run/cni/

2.8 RunContainerError

RunContainerError means the container runtime (containerd/docker) failed to start the container process. This is distinct from a container that starts and crashes — the runtime itself failed to execute the entrypoint.

Common causes: the entrypoint binary does not exist in the image (wrong path), the container user lacks permission to execute the file, or a required kernel capability is blocked by the SecurityContext or seccomp profile.

# Check the exact error
kubectl describe pod <POD_NAME> -n <NAMESPACE>
# Events: "Error: failed to create containerd task: ... exec: /app/start.sh: permission denied"

# Inspect the image entrypoint
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].command}'
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].args}'

# Check SecurityContext constraints
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.securityContext}'
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].securityContext}'

3. Service and Traffic Troubleshooting

3.1 Service Exists but No Traffic Reaches Pods

The most common cause: the Service has no Endpoints because the readinessProbe is failing. Kubernetes removes a pod from Service endpoints the moment its readiness probe fails — even if the pod is Running.

Commands to Run

# Step 1 — Check if Service has Endpoints
kubectl get endpoints <SERVICE_NAME> -n <NAMESPACE>
kubectl get ep <SERVICE_NAME> -n <NAMESPACE>
# Empty <none> = no pods match the selector

# Step 2 — Check EndpointSlices (Kubernetes 1.21+)
kubectl get endpointslices -n <NAMESPACE> -l kubernetes.io/service-name=<SERVICE_NAME>

# Step 3 — Verify Service selector matches Pod labels
kubectl get svc <SERVICE_NAME> -n <NAMESPACE> -o jsonpath='{.spec.selector}'
kubectl get pods -n <NAMESPACE> -l app=<LABEL_VALUE>

# Step 4 — Check readinessProbe — failing probe removes pod from endpoints
kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A 10 "Readiness"
# Look for: Readiness probe failed

# Step 5 — Test Service connectivity from another pod
kubectl exec -it <DEBUG_POD> -n <NAMESPACE> -- curl <SERVICE_NAME>.<NAMESPACE>.svc.cluster.local:<PORT>

# Step 6 — Check targetPort matches container port
kubectl get svc <SERVICE_NAME> -n <NAMESPACE> -o yaml | grep -E "port:|targetPort:"
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].ports}'

3.2 DNS Not Resolving

DNS failures in Kubernetes manifest as connection timeouts that look like network failures. The first thing to rule out is whether the issue is DNS or actual network connectivity.

Commands to Run

# Step 1 — Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Step 2 — Test DNS resolution from a pod
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- nslookup kubernetes.default.svc.cluster.local
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- nslookup <SERVICE_NAME>.<NAMESPACE>.svc.cluster.local

# Step 3 — Run a dedicated DNS debug pod
kubectl run dns-debug --image=busybox:1.28 --restart=Never -it --rm -- nslookup kubernetes

# Step 4 — Check resolv.conf inside a pod
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- cat /etc/resolv.conf
# Should show: search <NAMESPACE>.svc.cluster.local svc.cluster.local cluster.local
# ndots:5 is default — triggers 5 search domain expansions per query

# Step 5 — Check CoreDNS ConfigMap for misconfigs
kubectl get configmap coredns -n kube-system -o yaml

# Step 6 — Check CoreDNS metrics for saturation
kubectl top pod -n kube-system -l k8s-app=kube-dns
# High CPU on CoreDNS = likely DNS storm from ndots:5

3.3 503 Errors

A 503 from a Kubernetes Service or Ingress means the proxy (kube-proxy, nginx-ingress, Envoy) found no healthy backends. Cause: the pod was terminated or its readinessProbe started failing. The endpoint was removed from the EndpointSlice, leaving the load balancer with no backends. Fix: check pod readiness, verify the deployment rollout did not leave zero available replicas, and add minReadySeconds to your deployment to prevent premature endpoint registration.

3.4 504 Gateway Timeout

The load balancer or Ingress controller reached a backend but the backend did not respond within the configured timeout. Common causes: the application is CPU-throttled and processing slowly, a database query is running long, or the readTimeout on the Ingress annotation is set lower than the actual response time. Check Ingress annotations: nginx.ingress.kubernetes.io/proxy-read-timeout.

3.5 Connection Refused

Connection refused at the TCP level means the port is not listening. The most common cause is a targetPort mismatch — the Service targetPort points to port 8080 but the container is actually listening on port 3000. Verify: kubectl get svc -o yaml and confirm targetPort matches the container's declared containerPort and what the process is actually bound to (exec into the pod and run netstat -tlnp or ss -tlnp).

4. Node Troubleshooting

4.1 NotReady Nodes

A NotReady node means the kubelet is not reporting to the API server. This can be because the kubelet process is down, the node lost network connectivity (network partition), or the node is under extreme resource pressure.

Commands to Run

# Step 1 — Get all nodes and their status
kubectl get nodes -o wide

# Step 2 — Describe a NotReady node
kubectl describe node <NODE_NAME>
# Look at Conditions section:
#   Type              Status
#   MemoryPressure    False    (True = evictions will happen)
#   DiskPressure      False    (True = pods will be evicted)
#   PIDPressure       False
#   Ready             True     (False = kubelet is down or network partition)

# Step 3 — Check node resource allocation
kubectl describe node <NODE_NAME> | grep -A 10 "Allocated resources"
# Shows: cpu/memory requested vs allocatable

# Step 4 — Check for cordoned nodes
kubectl get nodes | grep SchedulingDisabled

# Step 5 — Check eviction events
kubectl get events -n <NAMESPACE> --field-selector reason=Evicted --sort-by='.lastTimestamp'

# Step 6 — Debug directly on node filesystem
kubectl debug node/<NODE_NAME> -it --image=ubuntu -- bash
# Inside: check disk: df -h, check memory: free -m, check kubelet: systemctl status kubelet

# Step 7 — SSH to node and check kubelet logs (if cloud provider allows)
journalctl -u kubelet -f --since "10 minutes ago"

4.2 MemoryPressure and Evictions

When a node reaches the memory eviction threshold (evictionHard: memory.available < 100Mi by default), the kubelet begins evicting pods in order of: BestEffort (no requests/limits) first, then Burstable (requests != limits), then Guaranteed last. Evictions appear as events with reason Evicted.

# Find evicted pods
kubectl get pods -n <NAMESPACE> --field-selector status.phase=Failed | grep Evicted

# See eviction reason
kubectl describe pod <EVICTED_POD> -n <NAMESPACE>
# Message: "The node was low on resource: memory. Threshold quantity: 100Mi..."

# Check node memory pressure
kubectl describe node <NODE_NAME> | grep -A 2 MemoryPressure

# Find which pods are using the most memory on the node
kubectl top pods -A --field-selector spec.nodeName=<NODE_NAME> --sort-by=memory

4.3 DiskPressure and Cascading Evictions

DiskPressure is triggered when the node's available disk space drops below the eviction threshold. Pods writing large amounts to their container filesystem (ephemeral storage) are the most common cause. The kubelet evicts pods consuming the most ephemeral storage first.

# Check disk pressure condition
kubectl describe node <NODE_NAME> | grep -A 2 DiskPressure

# Debug node disk usage
kubectl debug node/<NODE_NAME> -it --image=ubuntu -- bash
# Then inside the debug pod (chrooted to node filesystem):
df -h /host
du -sh /host/var/lib/containerd/*      # container layers
du -sh /host/var/log/pods/*            # pod logs
du -sh /host/var/lib/kubelet/pods/*    # pod volumes

# Check pod ephemeral storage limits
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].resources}'

4.4 Node Cordoned or Drained Unintentionally

# Check for cordoned nodes
kubectl get nodes | grep SchedulingDisabled

# Uncordon a node
kubectl uncordon <NODE_NAME>

# See who/what cordoned the node (check recent events and audit logs)
kubectl describe node <NODE_NAME> | grep -A 5 "Taints"
# node.kubernetes.io/unschedulable:NoSchedule = cordoned

5. Resource Troubleshooting

5.1 CPU Throttling

CPU throttling is the sneakiest performance problem in Kubernetes. The pod is not OOMKilled, it is Running, but the application is slow or timing out. The Linux CFS (Completely Fair Scheduler) enforces CPU limits by throttling the process when it exceeds its quota in a 100ms window.

Commands to Run

# CPU throttling — NOT visible as OOMKill, shows as slow application
# Step 1 — Check CPU limits vs requests ratio
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{range .spec.containers[*]}{.name}{": cpu requests="}{.resources.requests.cpu}{" limits="}{.resources.limits.cpu}{"
"}{end}'

# Step 2 — Check if pod is being throttled (requires cAdvisor or metrics)
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- cat /sys/fs/cgroup/cpu/cpu.stat
# Look for: throttled_time — if non-zero and growing, container is CPU-throttled

# Step 3 — Use top to see live CPU
kubectl top pod -n <NAMESPACE> --containers
# If usage is exactly at limit and app is slow = CPU throttled

# Memory leak detection
# Step 4 — Watch memory growth over time
watch kubectl top pod <POD_NAME> -n <NAMESPACE> --containers
# Steadily growing memory without OOMKill = memory leak, limits set too high

# Step 5 — Check ephemeral storage usage (disk inside container)
kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A 5 "ephemeral-storage"

5.2 Memory Leaks

# Watch memory trend — run every 30 seconds for 5 minutes
for i in $(seq 1 10); do
  echo "--- $(date) ---"
  kubectl top pods -n <NAMESPACE> --containers
  sleep 30
done

# If memory grows monotonically without GC pauses flattening it = memory leak

# For Go services: expose pprof, port-forward, profile
kubectl port-forward <POD_NAME> -n <NAMESPACE> 6060:6060
# Then: go tool pprof http://localhost:6060/debug/pprof/heap

# For JVM: get heap dump
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- jmap -dump:format=b,file=/tmp/heap.hprof <PID>
kubectl cp <NAMESPACE>/<POD_NAME>:/tmp/heap.hprof ./heap.hprof

5.3 PVC Full

# Check PVC usage (requires metrics-server with PVC metrics, or exec into pod)
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- df -h /data

# Check PVC capacity and access mode
kubectl get pvc -n <NAMESPACE>
kubectl describe pvc <PVC_NAME> -n <NAMESPACE>

# Resize PVC (if StorageClass supports volume expansion)
kubectl patch pvc <PVC_NAME> -n <NAMESPACE>   -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
# Note: the StorageClass must have allowVolumeExpansion: true

6. Network Troubleshooting

Network Debugging Path

  Can Pod A reach Service B?
          │
          ▼
  nslookup <svc>.<ns>.svc.cluster.local  (from pod exec)
  ├── FAILS ──▶  DNS issue
  │              kubectl get pods -n kube-system -l k8s-app=kube-dns
  │              kubectl logs -n kube-system <coredns-pod>
  │              Check: ndots setting, CoreDNS ConfigMap
  │
  └── RESOLVES ──▶  Can you curl the ClusterIP?
                    ├── NO  ──▶  kube-proxy / iptables issue
                    │            kubectl get ep <svc> -n <ns>  → empty?
                    │            Check readinessProbe on target pods
                    │
                    └── YES ──▶  Getting correct response?
                                 ├── WRONG BACKEND ──▶  Label selector mismatch
                                 │                      kubectl get svc -o yaml
                                 │
                                 └── TIMEOUT/REFUSED ──▶  NetworkPolicy blocking?
                                                          kubectl get netpol -n <ns>
                                                          Check ingress/egress rules

6.1 Pod Cannot Reach Service

# Step 1 — From inside the source pod
kubectl exec -it <SOURCE_POD> -n <NAMESPACE> -- sh

# Test DNS resolution
nslookup <TARGET_SERVICE>.<TARGET_NAMESPACE>.svc.cluster.local

# Test TCP connectivity
curl -v http://<TARGET_SERVICE>.<TARGET_NAMESPACE>.svc.cluster.local:<PORT>

# Test using the ClusterIP directly (bypass DNS)
kubectl get svc <TARGET_SERVICE> -n <TARGET_NAMESPACE> -o jsonpath='{.spec.clusterIP}'
curl -v http://<CLUSTER_IP>:<PORT>

# Step 2 — If DNS works but ClusterIP unreachable, check kube-proxy
kubectl get pods -n kube-system | grep kube-proxy
kubectl logs -n kube-system <KUBE_PROXY_POD>

# Step 3 — Verify iptables rules on node (if kube-proxy mode is iptables)
kubectl debug node/<NODE_NAME> -it --image=ubuntu -- nsenter -t 1 -m -u -i -n --   iptables -t nat -L KUBE-SERVICES | grep <SERVICE_NAME>

6.2 NetworkPolicy Blocking Traffic

# Step 1 — List all NetworkPolicies in namespace
kubectl get networkpolicy -n <NAMESPACE>
kubectl describe networkpolicy <POLICY_NAME> -n <NAMESPACE>

# Step 2 — Check if policy selects your pod
kubectl get networkpolicy -n <NAMESPACE> -o jsonpath='{range .items[*]}{.metadata.name}{": podSelector="}{.spec.podSelector}{"
"}{end}'

# Step 3 — Test connectivity to diagnose which direction is blocked
# Egress test from pod A:
kubectl exec -it <POD_A> -n <NAMESPACE> -- curl -v http://<POD_B_IP>:8080
# Ingress test — try from another pod
kubectl exec -it <POD_B> -n <NAMESPACE_B> -- curl -v http://<SVC_NAME>.<NAMESPACE>:80

# Step 4 — Temporarily allow all traffic to confirm NetworkPolicy is the issue
# (test only — revert immediately)
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-temp
  namespace: <NAMESPACE>
spec:
  podSelector: {}
  ingress:
  - {}
  egress:
  - {}
  policyTypes:
  - Ingress
  - Egress
EOF

# Step 5 — Use Cilium Hubble (if using Cilium CNI) for flow visibility
hubble observe --namespace <NAMESPACE> --verdict DROPPED

6.3 DNS Latency Spikes

In clusters with many pods, the ndots:5 default setting causes every hostname lookup to try up to 5 search domain expansions before falling through to the absolute name. A pod calling api.external.com will first try api.external.com.namespace.svc.cluster.local, then api.external.com.svc.cluster.local, then api.external.com.cluster.local, and two more variants before resolving the real hostname. At scale, this multiplies DNS query volume dramatically.

# Measure DNS query rate on CoreDNS
kubectl top pod -n kube-system -l k8s-app=kube-dns

# Check CoreDNS metrics (if prometheus is installed)
kubectl port-forward -n kube-system svc/kube-dns 9153:9153
# curl http://localhost:9153/metrics | grep coredns_dns_requests_total

# Fix ndots for a deployment — add dnsConfig
# In pod spec:
# dnsConfig:
#   options:
#   - name: ndots
#     value: "2"
# This drastically reduces search domain expansion for external names

# Alternatively, use a trailing dot to force absolute resolution:
# curl http://api.external.com.    # the trailing dot = absolute, no search domains

7. Control Plane Troubleshooting

7.1 kubectl Commands Hanging

# Step 1 — kubectl hangs or is very slow
kubectl get nodes --request-timeout=10s
# Timeout = API server unreachable or overloaded

# Step 2 — Check API server health
curl -k https://<API_SERVER_IP>:6443/healthz
curl -k https://<API_SERVER_IP>:6443/readyz

# Step 3 — Check control plane pods (kubeadm clusters)
kubectl get pods -n kube-system | grep -E "etcd|kube-apiserver|kube-controller|kube-scheduler"

# Step 4 — Check etcd health
kubectl exec -it etcd-<NODE> -n kube-system -- etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key   endpoint health

# Step 5 — Check etcd disk latency (etcd is extremely sensitive to disk I/O)
kubectl exec -it etcd-<NODE> -n kube-system -- etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key   check perf

# Step 6 — Check kube-controller-manager logs
kubectl logs -n kube-system kube-controller-manager-<NODE> --tail=100 | grep -i error

# Step 7 — API server audit logs for slow requests
kubectl logs -n kube-system kube-apiserver-<NODE> --tail=200 | grep -E "slow|timeout|error"

7.2 API Server Slow Responses

# Check API server request latency metric (if prometheus installed)
# apiserver_request_duration_seconds_bucket

# Identify expensive LIST operations
kubectl logs -n kube-system kube-apiserver-<NODE> --tail=500 |   grep -E "took too long|slow" | sort | tail -20

# Check for excessive watchers (too many controllers or informers)
kubectl get --raw /metrics | grep apiserver_registered_watchers

# Check etcd object counts (large clusters can have etcd size issues)
kubectl exec -it etcd-<NODE> -n kube-system -- etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key   endpoint status -w table

8. Full Debugging Command Reference

Events — Your Most Important Tool

# Get all events sorted by time (most recent last)
kubectl get events -n <NAMESPACE> --sort-by='.lastTimestamp'

# Get only Warning events
kubectl get events -n <NAMESPACE> --field-selector type=Warning --sort-by='.lastTimestamp'

# Get events for a specific pod
kubectl get events -n <NAMESPACE> --field-selector involvedObject.name=<POD_NAME>

# Get events across ALL namespaces
kubectl get events -A --sort-by='.lastTimestamp' | tail -30

# Watch events in real time
kubectl get events -n <NAMESPACE> -w

# Get events with full details
kubectl describe events -n <NAMESPACE>

Quick Diagnostic One-Liners

# Show all non-Running pods in a namespace
kubectl get pods -n <NAMESPACE> --field-selector status.phase!=Running

# Show all pods with high restart counts
kubectl get pods -A | awk 'NR>1 {if ($5+0 > 5) print $0}'

# Show resource usage for all pods in a namespace
kubectl top pods -n <NAMESPACE> --containers --sort-by=memory

# Show all pods on a specific node
kubectl get pods -A --field-selector spec.nodeName=<NODE_NAME>

# Show all recent Warning events cluster-wide
kubectl get events -A --field-selector type=Warning --sort-by='.lastTimestamp' | tail -30

# Check deployment rollout status
kubectl rollout status deployment/<DEPLOY_NAME> -n <NAMESPACE>

# Check rollout history
kubectl rollout history deployment/<DEPLOY_NAME> -n <NAMESPACE>

# Roll back a bad deployment
kubectl rollout undo deployment/<DEPLOY_NAME> -n <NAMESPACE>

# Force a new rollout (triggers pod replacement without changing spec)
kubectl rollout restart deployment/<DEPLOY_NAME> -n <NAMESPACE>

# Check all resources in a namespace
kubectl get all -n <NAMESPACE>

# Get a pod&apos;s full spec with current status
kubectl get pod <POD_NAME> -n <NAMESPACE> -o yaml

# Watch pods in real time
kubectl get pods -n <NAMESPACE> -w

# Stream logs from all pods in a deployment
kubectl logs -n <NAMESPACE> -l app=<LABEL> -f --max-log-requests=10

# Check which image version is running
kubectl get pods -n <NAMESPACE> -o jsonpath='{range .items[*]}{.metadata.name}{": "}{range .spec.containers[*]}{.image}{"
"}{end}{end}'

9. Three Production Incident Stories

10. Common Troubleshooting Mistakes

1. Reading logs before checking events. Events expire in 1 hour; logs persist longer. But events have the scheduler's exact reason — read events first.
2. Checking kubectl top node for scheduling capacity. Top shows actual usage; scheduling uses requested resources. A node can be 10% utilized but fully scheduled.
3. Assuming Running = healthy. A pod can be Running but not Ready. It will be excluded from Service endpoints. Always check READY column: 1/1 vs 0/1.
4. Force-deleting StatefulSet pods without node verification. Causes split-brain. Only force-delete after confirming the node is completely offline.
5. Increasing memory limits to fix OOMKill without checking for leaks. If it is a leak, you are just delaying the next OOMKill.
6. Setting CPU limits on latency-sensitive workloads. CPU throttling causes unpredictable latency. Consider removing CPU limits and only setting requests.
7. Not checking LimitRange when pods go Pending. The most commonly forgotten resource constraint in multi-team clusters.
8. Debugging DNS by pinging. Ping uses ICMP, not DNS+TCP. Use nslookup or dig or curl to properly test DNS resolution.
9. Assuming NetworkPolicy allows DNS. A restrictive egress policy that does not explicitly allow UDP/TCP port 53 to kube-dns will silently break all service discovery.
10. Not using --previous flag when container is restarting. kubectl logs without --previous shows the current container's logs — which may be empty if it just started. The crash details are in the previous container.
11. Ignoring init container logs. If a pod is stuck at Init:0/1, the main container logs are empty. The failure is in the init container.
12. Not accounting for termination grace period. A pod in Terminating with a 30-second grace period is expected behavior. Only investigate after it exceeds the grace period significantly.
13. Checking the wrong namespace. Always confirm your kubectl context and namespace. Add -n NAMESPACE explicitly. kubectl config set-context --current --namespace=NAMESPACE can help, but also cause confusion.
14. Assuming the issue is in the most recently changed component. Cascading failures often originate elsewhere. A node disk filling slowly for hours can cause sudden evictions that look like a deployment problem.
15. Not using -o wide when getting pods. Without -o wide, you cannot see which node the pod is on — critical for node-level failures and affinity debugging.
16. Forgetting that HPA scale-down can cause temporary unavailability. If HPA scales down during high traffic, the removed pods are real. Check HPA status during traffic incidents.

11. Interview Questions and Answers

Beginner Questions

Q1: A pod is in Pending state. What do you check first?

A: Run kubectl describe pod <NAME> and look at the Events section at the bottom. The scheduler writes the exact reason: insufficient resources, taint mismatch, affinity rules, unbound PVC, or quota exceeded. Each reason points to a different fix. Without the event message, you are guessing.

Q2: How do you get logs from a pod that is CrashLoopBackOff?

A: Use kubectl logs <POD> --previous. The --previous flag retrieves logs from the last terminated container instance. Without it, you get the current container's logs, which may be nearly empty if it just started. The exit code in kubectl describe pod under "Last State" tells you the category of failure before you even read the logs.

Q3: What does CrashLoopBackOff mean? Why does it back off?

A: The container is repeatedly starting and crashing. Kubernetes applies exponential backoff (10s, 20s, 40s, up to 5 minutes) between restart attempts to prevent a crashing pod from consuming excessive CPU and creating excessive API server events. The back-off is per-container. If the container runs successfully for 10 minutes, the back-off resets.

Q4: What is the difference between liveness and readiness probes?

A: A readiness probe failure removes the pod from Service endpoints — traffic stops being sent to it, but the pod keeps running. A liveness probe failure causes the kubelet to kill and restart the container. Readiness is for "not ready to serve traffic yet" (warm-up, dependency not available). Liveness is for "stuck in an unrecoverable state" (deadlock, hung process). Misconfiguring liveness probes with too-short initialDelaySeconds causes healthy containers to be killed during startup.

Q5: How do you find which pods are using the most memory in a namespace?

A: kubectl top pods -n <NAMESPACE> --containers --sort-by=memory. This requires the metrics-server to be installed. For historical data, query Prometheus/Grafana.

Intermediate Questions

Q6: A Service has no endpoints even though its pods are Running. Why?

A: Three possible causes. First, the Service selector does not match any pod labels — verify with kubectl get svc -o yaml and compare the selector to actual pod labels. Second, the pods are Running but not Ready — readinessProbe is failing, which removes them from endpoints. Check kubectl describe pod for readiness probe failures. Third, the pods are in a different namespace from the Service — Services only select pods in the same namespace.

Q7: How do you debug DNS resolution inside a pod?

A: Exec into the pod: kubectl exec -it <POD> -- nslookup kubernetes.default.svc.cluster.local. If that fails, check CoreDNS pods are Running in kube-system. Check /etc/resolv.conf inside the pod to confirm the nameserver points to the kube-dns ClusterIP. Check CoreDNS logs for errors. If CoreDNS is overloaded (100% CPU), DNS queries time out intermittently.

Q8: What is a finalizer and why can it prevent pod deletion?

A: A finalizer is a key in metadata.finalizers that instructs Kubernetes to run cleanup logic before deleting the object. When you delete a pod with finalizers, Kubernetes sets deletionTimestamp but does not remove the object until all finalizers are cleared. If the controller responsible for clearing the finalizer is down or buggy, the pod stays Terminating forever. Fix: patch the finalizers to an empty array, which immediately allows deletion to proceed.

Q9: What is the difference between OOMKilled and a pod being evicted?

A: OOMKilled is a container-level event — the container exceeded its cgroup memory limit, and the Linux OOM killer terminated its process. The pod restarts. Eviction is a pod-level event triggered by the kubelet when the node is under memory or disk pressure. The kubelet actively terminates pods (starting with BestEffort quality class) to free resources on the node. An evicted pod does not automatically restart on the same node; the scheduler places it on another node.

Q10: How do you diagnose CPU throttling in Kubernetes?

A: CPU throttling is not visible in kubectl top alone — it only shows actual CPU consumption. To detect throttling, exec into the container and read /sys/fs/cgroup/cpu/cpu.stat (cgroups v1) or /sys/fs/cgroup/cpu.stat (cgroups v2) and look at the throttled_time field. If it is non-zero and growing, the container is being throttled. In Prometheus, query rate(container_cpu_cfs_throttled_seconds_total[5m]).

Advanced Questions

Q11: A node is NotReady and kubectl commands are timing out. What do you do?

A: First, verify the cluster API server itself is healthy by running kubectl get nodes --request-timeout=10s from a different context or client. If the API server responds, the issue is isolated to that node. Set the node to unschedulable (kubectl cordon) to prevent new pods from scheduling there. Attempt to SSH to the node and check kubelet: systemctl status kubelet and journalctl -u kubelet -f. Check for disk pressure (df -h) and memory pressure (free -m). If the node is in a cloud provider, check the provider console for hardware failures. Force-drain the node to migrate workloads: kubectl drain <NODE> --ignore-daemonsets --delete-emptydir-data.

Q12: etcd is having leader election issues. What causes this and how do you diagnose it?

A: etcd leader elections are most commonly caused by high disk I/O latency. etcd uses raft consensus and requires disk writes (fsync) before acknowledging a write. If disk latency exceeds the election timeout (default 1 second), followers cannot send heartbeats in time and trigger an election. Diagnose with etcdctl check perf, which measures sequential write latency. Also check etcdctl endpoint status to see if there are multiple leaders (split-brain) or frequent term increments. Other causes: network partition between etcd members, and clock skew between nodes.

Q13: How do you investigate a NetworkPolicy that is blocking unexpected traffic?

A: Start by listing all NetworkPolicies in both the source and destination namespaces. Remember: a NetworkPolicy applies to pods selected by its podSelector, and controls both ingress and egress depending on policyTypes. Test connectivity systematically — exec into the source pod, try to reach the destination by ClusterIP (bypass DNS), then by Service name. If Cilium is the CNI, use Hubble to observe dropped flows with policy reasons. Temporarily apply an allow-all policy to confirm NetworkPolicy is the blocker before narrowing down which rule.

Q14: A Deployment rollout is stuck at "Waiting for deployment spec update to be observed." What does this mean?

A: The kube-controller-manager has not yet processed the updated Deployment spec. This usually indicates kube-controller-manager is lagging, overloaded, or has lost its leader election. Check the kube-controller-manager pod: kubectl logs -n kube-system kube-controller-manager-<NODE>. If you see leader election errors or rate-limiting messages, the controller plane is under stress. Also check the API server QPS limits — if the controller manager is being rate-limited, it will lag behind processing updates.

Q15: How would you architect a Kubernetes cluster to be production-ready from a reliability perspective?

A: Multi-zone control plane (3 etcd members across 3 AZs). Separate etcd nodes with NVMe SSDs and dedicated low-latency disks. Node pools per workload type (CPU-intensive, memory-intensive, spot/preemptible). PodDisruptionBudgets on all critical workloads to survive node drains. HPA plus VPA for automatic scaling. Resource requests and limits (with careful consideration on CPU limits for latency-sensitive services). PodTopologySpreadConstraints for zone-aware scheduling. Comprehensive NetworkPolicy. Audit logging on the API server. etcd backup to object storage every 30 minutes with tested restore procedure. Cluster-level Prometheus with alerts on: node NotReady, pod CrashLoopBackOff rate, PVC near-full, etcd disk latency, API server error rate, and certificate expiry.

12. Twenty Troubleshooting Best Practices

1. Always define resource requests for every container. Scheduling without requests is a gamble.
2. Set memory limits to prevent containers from consuming all node memory and triggering evictions.
3. Consider not setting CPU limits on latency-sensitive services to avoid throttling.
4. Always define readiness probes. Without them, pods receive traffic before they are ready.
5. Set ephemeral-storage limits to prevent disk-filling containers from causing cascading evictions.
6. Write application logs to stdout/stderr, not to the container filesystem.
7. Use PodDisruptionBudget for every production workload to maintain availability during cluster operations.
8. Set minReadySeconds on Deployments to prevent routing traffic to pods before they are fully warm.
9. Use terminationGracePeriodSeconds that is longer than your slowest graceful shutdown operation.
10. Use preStop hooks with a short sleep (5–10 seconds) to allow the load balancer to drain connections before the pod terminates.
11. Mirror LimitRange and ResourceQuota from production to staging environments.
12. Add dnsConfig.options: ndots: 2 for services that frequently call external APIs.
13. Enable NodeLocal DNSCache on all nodes to reduce CoreDNS load.
14. Enforce ephemeral-storage limits cluster-wide via OPA/Gatekeeper or Kyverno policy.
15. Keep etcd on dedicated nodes with NVMe storage. Never co-locate etcd with high-I/O workloads.
16. Back up etcd every 30 minutes and test the restore procedure quarterly.
17. Use PodTopologySpreadConstraints to distribute pods across zones and nodes.
18. Set HPA.spec.behavior.scaleDown.stabilizationWindowSeconds to prevent premature scale-down during brief traffic dips.
19. Alert on certificate expiry at 30 days, 14 days, and 7 days — expired certificates cause instant cluster-wide failures.
20. Label all resources consistently to enable fast filtering during incidents: app, version, component, environment.

13. Frequently Asked Questions

Why is my pod Running but receiving no traffic?

The readiness probe is failing. A Running pod is excluded from Service endpoints when its readiness probe fails. Check: kubectl describe pod for readiness probe failure events, and kubectl get ep <SERVICE> to confirm the pod IP is absent from endpoints.

How long should I wait before force-deleting a Terminating pod?

Wait at least twice the terminationGracePeriodSeconds (default 30 seconds, so at least 60 seconds). If the pod is still terminating, check for finalizers first. Force delete only as a last resort, and never on StatefulSet pods without verifying node status.

What is the difference between kubectl delete pod --force and removing finalizers?

--force --grace-period=0 removes the pod object from etcd immediately but the process may still run on the node. Removing finalizers allows the normal deletion flow to proceed — the pod object stays until the kubelet confirms the container is stopped. Removing finalizers is always safer.

Why does my pod keep getting evicted?

The node is under memory or disk pressure. The pod is likely BestEffort class (no resource requests) and is the first eviction target. Add resource requests to make the pod Burstable or Guaranteed class. Also investigate the root cause of the pressure — another pod may be consuming excessive resources.

How do I debug a pod that crashes too fast to exec into?

Use kubectl debug to create a copy with a modified command: kubectl debug <POD> -it --copy-to=debug-pod --container=<CONTAINER> -- sleep 3600. This creates a running copy where you can exec in and inspect the filesystem, environment variables, and connectivity without the crashing entrypoint running.

Why is DNS resolution slow intermittently in my cluster?

Most commonly: CoreDNS is saturated by ndots:5 search domain expansion. Also check: connection tracking table overflow on nodes (the Linux conntrack table has a default limit, high-traffic clusters can exhaust it, causing new connections including DNS to be dropped). Check dmesg | grep nf_conntrack on nodes.

What is the difference between Guaranteed, Burstable, and BestEffort QoS?

Guaranteed: every container has equal requests and limits for both CPU and memory. Burstable: at least one container has a request less than its limit. BestEffort: no containers have requests or limits. The kubelet evicts BestEffort first, then Burstable by usage-to-request ratio, then Guaranteed last.

How do I see which containers are inside a pod?

kubectl get pod <POD> -n <NS> -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'

Why does kubectl exec hang?

The API server proxies exec connections through the kubelet on the node. If the node is under pressure or the kubelet is slow, exec connections hang. Also check that the API server and kubelet have compatible versions. Another cause: the container process is unresponsive (exec requires the container to be Running and accepting connections through the container runtime).

How do I check if HPA is scaling correctly?

kubectl get hpa -n <NAMESPACE>
kubectl describe hpa <HPA_NAME> -n <NAMESPACE>

The describe output shows: current metric value, target value, min/max replicas, and the last scale event reason. If HPA is not scaling, check that metrics-server is running and that the pods have CPU/memory requests defined (HPA uses requests as the baseline for utilization percentage).

Why are my pods not spreading evenly across nodes?

The default scheduler uses a scoring algorithm that considers available resources, not just pod count. Add topologySpreadConstraints with maxSkew: 1 and topologyKey: kubernetes.io/hostname to enforce even pod distribution across nodes.

What is the safest way to drain a node for maintenance?

kubectl cordon <NODE>  # Prevent new pods from scheduling
kubectl drain <NODE> --ignore-daemonsets --delete-emptydir-data --grace-period=60

Always cordon before draining so no new pods land on the node while you are migrating existing pods. The drain command respects PodDisruptionBudgets, which is why you should always have PDBs defined.

How do I check if a Secret or ConfigMap is actually mounted correctly in a pod?

kubectl exec -it <POD> -n <NS> -- ls /path/to/mount
kubectl exec -it <POD> -n <NS> -- env | grep <KEY>

What happens to running pods when the API server goes down?

Running pods continue to run. The kubelet and kube-proxy operate locally on each node and do not require the API server to maintain running containers or iptables rules. However, new pods cannot be scheduled, existing pods cannot be restarted if they crash, and all kubectl commands will fail. This is why control plane HA is critical but a brief API server outage does not immediately affect running traffic.

How do I check if a Deployment has been paused?

kubectl get deployment <NAME> -n <NS> -o jsonpath='{.spec.paused}'

A paused deployment will not roll out changes. Resume with: kubectl rollout resume deployment/<NAME> -n <NS>.

14. Key Takeaways

• Mental model over commands. The top-down hierarchy (Namespace → Workload → Pod → Container → Node → Network → Storage) systematically eliminates possibilities. Scope first, diagnose second.
• Events are your first stop. kubectl get events --sort-by=lastTimestamp gives you the scheduler's and kubelet's exact failure reason. Events expire in 1 hour — get them immediately.
• Exit codes are fingerprints. 137=OOMKilled, 1=app error, 127=command not found, 143=SIGTERM. Each points to a different fix.
• Running does not mean healthy. A Running pod with a failing readiness probe receives zero traffic. Always check the READY column and kubectl get ep.
• Requests drive scheduling, limits drive killing. CPU and memory limits determine throttling and OOMKills. Requests determine where the scheduler places pods.
• LimitRange is the invisible wall. In multi-team clusters, LimitRange requirements silently reject pods with no resource requests. Always check it when pods go Pending.
• ndots:5 scales poorly. In large clusters, the default DNS search domain setting multiplies DNS query volume 5x. Apply ndots: 2 to deployments that call external services.
• Never force-delete StatefulSet pods on live nodes. It causes split-brain. Force delete only after verifying node is completely offline.
• etcd is the heart of the cluster. Disk latency kills etcd. Run it on NVMe, monitor its write latency daily, and test your backup restore procedure quarterly.
• Write a runbook after every incident. The engineer in the intro resolved a 12-minute incident that could have taken hours. The difference was a mental model. Document yours.