Kubernetes Pod Lifecycle Explained: Phases, Conditions, Probes, and Which Pod HPA Kills First

kubectl apply
↓ Pod object created in etcd
↓ Scheduler selects a node
↓ kubelet starts the pod
↓ Init containers run (one by one)
↓ Main container starts
↓ Startup probe passes
↓ Readiness probe passes
↓ Pod receives traffic
↓ Liveness probe runs continuously
↓ Pod is deleted / crashes / restarted
↓ Kubernetes handles termination or restart

1. Pending
2. Running
3. Succeeded
4. Failed
5. Unknown

# Check pod status and phases
kubectl get pods -n <namespace>

# Describe pod for detailed info
kubectl describe pod <pod-name> -n <namespace>

# Check exact phase
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.phase}'

PodScheduled      → node has been assigned
Initialized       → all init containers completed
ContainersReady   → all containers are ready
Ready             → pod can receive traffic

kubectl describe pod <pod-name> -n <namespace>

# Look for:
# Conditions:
#   Initialized       True
#   Ready             False    ← not receiving traffic
#   ContainersReady   False
#   PodScheduled      True

# If Ready=False, the pod is not receiving traffic.
# Check why the readiness probe is failing.

Waiting     → container has not started yet
Running     → container process is running
Terminated  → container has exited

Pod lifecycle with init containers:
1. Init container 1 runs → must exit 0
2. Init container 2 runs → must exit 0
3. Main container starts

If any init container fails:
→ Pod stays in Pending
→ Initialized = False
→ Main container never starts

# Check init container status
kubectl describe pod <pod-name> -n <namespace>

# Look for:
# Init Containers:
#   wait-for-db:
#     State: Terminated (exit 0 = passed)
#     or
#     State: CrashLoopBackOff (it keeps failing)

# Read init container logs
kubectl logs <pod-name> -n <namespace> -c <init-container-name>

# If init container is failing:
# Status shows: Init:CrashLoopBackOff
# OR:           Init:0/2  (0 of 2 completed)

# Example init container in YAML
initContainers:
- name: wait-for-db
  image: busybox
  command: ['sh', '-c', 'until nc -z postgres-svc 5432; do sleep 2; done']

Pod
├── main container     (your API server)
├── sidecar: envoy     (handles traffic encryption)
└── sidecar: fluentd   (ships logs to Elasticsearch)

All three containers:
→ share the same IP address
→ communicate via localhost
→ can read/write the same mounted volumes

# Kubernetes 1.29+ native sidecar support
# Add restartPolicy: Always to an init container = it becomes a sidecar
# It starts before the main container and keeps running
# For Jobs: it does NOT block the Job from completing

initContainers:
- name: log-collector
  image: fluentd:v1.16
  restartPolicy: Always   # ← this makes it a native sidecar

1. Startup Probe   → Is the app done starting?
2. Readiness Probe → Can the app receive traffic?
3. Liveness Probe  → Is the app still alive (not stuck)?

Readiness failure  → removes traffic. Does NOT kill the container.
Liveness failure   → kills and restarts the container.
Startup failure    → kills and restarts. Blocks readiness + liveness.

# Complete production manifest — all three probes + lifecycle hooks
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: api
          image: my-api:2.0.0
          startupProbe:              # Runs first. Blocks readiness + liveness until it passes.
            httpGet: { path: /healthz/startup, port: 8080 }
            failureThreshold: 30     # 30 × 10s = up to 5 min to start
            periodSeconds: 10
          readinessProbe:            # Gates traffic. Removes from Service endpoints on failure.
            httpGet: { path: /healthz/ready, port: 8080 }
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3      # 15s of bad readiness before removed from endpoints
          livenessProbe:             # Kills and restarts on failure. Set threshold MUCH higher.
            httpGet: { path: /healthz/live, port: 8080 }
            periodSeconds: 10
            failureThreshold: 9      # 3× readiness — only fires on truly stuck processes
          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh", "-c", "curl -s -X POST http://consul:8500/v1/agent/service/register"]
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]  # closes iptables propagation window
          resources:
            requests: { memory: "256Mi", cpu: "100m" }
            limits:   { memory: "512Mi", cpu: "500m" }  # set -Xmx to 75% of limit for JVM

kubectl delete pod
↓ deletionTimestamp is set
↓ Pod is removed from Service endpoints (traffic stops)
↓ preStop hook runs (if configured)
↓ SIGTERM is sent to the container
↓ Grace period starts (default: 30 seconds)
↓ SIGKILL is sent if app does not stop
↓ Pod object is removed

# Both hooks in a single container spec
containers:
  - name: api
    image: my-api:2.0.0
    lifecycle:
      postStart:
        exec:
          # Runs right after container starts — in parallel with the app
          # Pod will NOT be marked Ready until this exits
          command: ["/bin/sh", "-c", "curl -s -X POST http://consul:8500/v1/agent/service/register -d @/config/service.json"]
      preStop:
        exec:
          # Runs BEFORE SIGTERM — pod stays alive until this exits
          # Use for cleanup: drain connections, deregister, flush queue
          command: ["/bin/sh", "-c", "sleep 5 && curl -s -X PUT http://consul:8500/v1/agent/service/deregister/my-api"]

# postStart failure kills and restarts the container
# preStop failure is ignored — SIGTERM fires anyway after grace period

# Check a pod's restart policy
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.restartPolicy}'

# Deployment default
spec:
  restartPolicy: Always   # always restart on any exit

# Job default
spec:
  restartPolicy: OnFailure  # restart only on non-zero exit

# Example:
Memory limit: 512Mi
App tries to use: 700Mi
Linux kernel kills the process → Kubernetes shows: OOMKilled
Exit code: 137

kubectl describe pod <pod-name> -n <namespace>

# Look for:
# Last State:
#   Reason:     OOMKilled
#   Exit Code:  137

# Check current memory usage
kubectl top pod <pod-name> -n <namespace>

# Check exact error in Events section
kubectl describe pod <pod-name> -n <namespace>

# Look at the Events: section at the bottom
# Failed to pull image "my-app:v2.0": rpc error...

resources:
  requests:
    memory: "256Mi"   # scheduler: find a node with 256Mi free
    cpu: "100m"       # scheduler: find a node with 100 millicores free
  limits:
    memory: "512Mi"   # OOMKilled if app uses more than 512Mi
    cpu: "500m"       # throttled (slowed) if app uses more than 500 millicores

# Check current resource usage
kubectl top pod <pod-name> -n <namespace>
kubectl top pod <pod-name> -n <namespace> --containers

# Check if CPU is throttled (on the node directly)
# cat /sys/fs/cgroup/cpu/kubepods/.../cpu.stat
# Look for: nr_throttled > 0

# Check a pod's QoS class
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.qosClass}'
# Output: BestEffort | Burstable | Guaranteed

# Guaranteed example (requests == limits for ALL containers)
resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"   # same as requests
    cpu: "500m"       # same as requests

Eviction triggers:
  MemoryPressure  → node memory is running low
  DiskPressure    → node disk is filling up
  PIDPressure     → too many processes on the node

Eviction order:
  1. BestEffort pods (no requests/limits) → evicted first
  2. Burstable pods (requests < limits) → evicted second
  3. Guaranteed pods (requests == limits) → evicted last

1. kubelet terminates the pod immediately (hard eviction = no grace period)
2. Pod object stays in etcd with STATUS: Evicted
3. ReplicaSet notices the pod is gone
4. ReplicaSet creates a new pod — on a DIFFERENT node
5. The evicted pod object stays until you clean it up manually

# Check if a pod was evicted
kubectl describe pod <pod-name> -n <namespace>
# Look for: Reason: Evicted
#           Message: The node was low on resource: memory

# Check node pressure conditions
kubectl describe node <node-name>
# Look for Conditions: MemoryPressure=True, DiskPressure=True

# Find all evicted pods
kubectl get pods -n <namespace> --field-selector=status.phase==Failed

# Clean up evicted pods
kubectl get pods -n <namespace> -o name   | xargs kubectl delete -n <namespace> --field-selector=status.phase==Failed

HPA detects: CPU usage is low
→ HPA updates spec.replicas from 10 to 6
→ ReplicaSet Controller decides which 4 pods to delete
→ kubelet terminates them

1. Unscheduled pod (never ran)     → dies first
2. Not Ready pod (probe failing)   → dies second
3. Newest Running pod              → dies third
4. Older Running pod               → dies later
5. Oldest Running pod              → survives longest

# Example: 4 replicas, maxSurge: 1, maxUnavailable: 1
# At most 5 pods can exist (4 + 1 surge)
# At most 1 pod can be unavailable — so at least 3 are always Ready

Rolling update sequence:
1. Create 1 new pod (surge) → total: 5 pods (4 old + 1 new)
2. Wait for new pod to pass readiness probe → it becomes Ready
3. Terminate 1 old pod → total: 4 pods (3 old + 1 new)
4. Create 1 new pod → total: 5 pods (3 old + 2 new)
5. Wait for new pod to be Ready
6. Terminate 1 old pod → total: 4 pods (2 old + 2 new)
7. Repeat until all old pods are replaced

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

# Monitor rollout progress
kubectl rollout status deployment/<name> -n <namespace>

# Rollback to previous version
kubectl rollout undo deployment/<name> -n <namespace>

# Rollback to a specific revision
kubectl rollout history deployment/<name> -n <namespace>
kubectl rollout undo deployment/<name> --to-revision=2 -n <namespace>

Deployment  → pods are identical. Any pod handles any request.
StatefulSet → pods have identity (pod-0, pod-1). Order matters.
DaemonSet   → one pod per node. Follows nodes, not replica count.

Scale-down:
  Deployment  → newest pod dies first
  StatefulSet → pod-2 dies first, pod-0 dies last
  DaemonSet   → no scale-down. Remove node = remove pod.

# Check for finalizers
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A5 finalizers

# Example output:
# metadata:
#   finalizers:
#   - example.com/my-finalizer   ← pod cannot delete until this is removed

# Force-remove all finalizers (use when the controller is gone)
kubectl patch pod <pod-name> -n <namespace>   -p '{"metadata":{"finalizers":[]}}' --type=merge

# Last resort: force delete without grace period
kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force

# Step 1: read the exit reason, not the logs
kubectl describe pod my-pod

# Look for this block under Containers → Last State:
#   Last State:     Terminated
#     Reason:       OOMKilled   ← or: Error, Signal, StartError
#     Exit Code:    137         ← 137=SIGKILL(OOM), 1=app error, 143=SIGTERM, 126/127=bad command
#     Started:      Mon, 24 Jun 2026 04:01:00 +0000
#     Finished:     Mon, 24 Jun 2026 04:01:00 +0000  ← same second = died instantly on startup

# If Started and Finished are the same second:
#   → container never ran your app — check the ENTRYPOINT/CMD
#   → check: is the binary missing? wrong file permissions? SecurityContext blocking exec?

# Step 2: check if a startupProbe is killing it before it writes logs
#   startupProbe with failureThreshold too low = container killed before app initializes

# Step 3: check SecurityContext
#   runAsNonRoot: true + image running as root → container exits immediately, no logs

# Step 4: check node-level events
kubectl get events --sort-by='.lastTimestamp' -n my-namespace

# Get pods in namespace
kubectl get pods -n <namespace>

# Describe pod (most useful command)
kubectl describe pod <pod-name> -n <namespace>

# Current container logs
kubectl logs <pod-name> -n <namespace>

# Previous crashed container logs (very important)
kubectl logs <pod-name> -n <namespace> --previous

# Check events sorted by time
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

# Check resource usage
kubectl top pod <pod-name> -n <namespace>

# Check exact pod phase
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.phase}'

# Check last container state (exit code + reason)
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState}'

# Exec into a running container
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

# Debug distroless or minimal containers
kubectl debug -it <pod-name> --image=busybox:1.28 --target=<container-name>