Kubernetes StatefulSet vs Deployment — Every Question Answered

A Deployment is the Kubernetes object you use to run stateless applications. When you create a Deployment, Kubernetes creates a ReplicaSet under the hood, and that ReplicaSet creates and manages N pods. All pods in a Deployment are anonymous and interchangeable — any pod can handle any request, and if one crashes, Kubernetes replaces it with a fresh pod with a new random name and a clean start.

Use Deployment for: web servers, REST APIs, GraphQL services, background workers, ML inference servers — anything that does not write state to local disk and does not care which specific pod handles which request.

# Create a Deployment
kubectl apply -f deployment.yaml

# What Kubernetes creates:
Deployment: my-api
  └── ReplicaSet: my-api-7d4b9c
        ├── Pod: my-api-7d4b9c-xk2pq  (random name)
        ├── Pod: my-api-7d4b9c-ab3df  (random name)
        └── Pod: my-api-7d4b9c-mp9tz  (random name)

A StatefulSet is the Kubernetes object you use to run stateful applications. Unlike a Deployment, every pod in a StatefulSet has a permanent identity — a fixed name based on its ordinal number (mysql-0, mysql-1, mysql-2), its own dedicated PersistentVolumeClaim, and a stable DNS hostname. This identity survives pod restarts.

If mysql-1 crashes and Kubernetes restarts it, the new pod is still named mysql-1, gets the same PVC (data-mysql-1) reattached, and has the same DNS hostname. Nothing in the cluster has to reconfigure. Data is preserved.

# Create a StatefulSet
kubectl apply -f statefulset.yaml

# What Kubernetes creates:
StatefulSet: mysql
  ├── Pod: mysql-0  → PVC: data-mysql-0  → DNS: mysql-0.mysql-headless.default.svc.cluster.local
  ├── Pod: mysql-1  → PVC: data-mysql-1  → DNS: mysql-1.mysql-headless.default.svc.cluster.local
  └── Pod: mysql-2  → PVC: data-mysql-2  → DNS: mysql-2.mysql-headless.default.svc.cluster.local

Deployment was built on a core assumption: all pods are identical and interchangeable. This works perfectly for stateless apps. But when teams tried to run databases (MySQL, PostgreSQL, MongoDB) as Deployments, they ran into a wall — a crashed pod came back with a new name, a new IP, and no memory of its old data. Replicas lost their replication source. Clusters became corrupted.

StatefulSet was introduced in Kubernetes 1.5 (reached stable in 1.9) specifically to fill this gap. It gives pods the three things Deployment cannot provide: a stable identity that survives restarts, a dedicated PVC per pod, and guaranteed ordered startup and shutdown. Deployment was not wrong — it was the right tool for 80% of workloads. StatefulSet handles the remaining 20%.

StatefulSet solves three problems that Deployment cannot:

• Stable identity: pod names never change (mysql-0 is always mysql-0), so other pods can always find it by the same DNS hostname.
• Per-pod storage: each pod gets its own PVC via volumeClaimTemplates. Data survives pod crashes and node changes.
• Ordered lifecycle: pods start in order (0 → 1 → 2, each waiting for the previous to be Ready) and stop in reverse order (2 → 1 → 0).

A stateless application does not store any data between requests. Every request is self-contained. Any pod can handle any request. If you delete a pod and start a new one, nothing is lost. State lives outside the app — in a database, a cache, or an external service.

A stateful application stores data on disk that must survive restarts. Each instance owns a specific slice of data. You cannot swap one instance for another because the replacement would not have the same data. Pod identity and storage continuity both matter.

A stateless application is one where each request is handled independently, with no reliance on local in-memory or on-disk state from a previous request. All persistent state is stored in an external system — a database, Redis, S3. Any pod can handle any request at any time. Horizontal scaling is trivial: add more pods.

Examples: nginx, express, Django, FastAPI, Spring Boot (when configured correctly), Rails.

A stateful application maintains data on disk across requests and across restarts. Each instance has a unique role — a primary database writes data that replicas read. Each Kafka broker owns specific partitions. Each Elasticsearch node holds specific shards. You cannot replace an instance arbitrarily — the replacement would start with a clean disk and break the cluster.

Examples: PostgreSQL, MySQL, MongoDB, Apache Kafka, Elasticsearch, Redis Cluster, ZooKeeper.

Web applications are designed stateless intentionally. Storing session data in a cookie or JWT (rather than in-process memory) means any pod can authenticate any request. Storing user data in a shared database means any pod can serve any user. This design choice makes horizontal scaling simple — add pods, remove pods, crash pods — the application keeps working. If web applications were stateful, every user would have to be pinned to a specific pod, which kills horizontal scalability.

Databases write data to disk. That is their entire purpose. The data a database stores cannot be reconstructed if the disk is lost — unlike a stateless app where you simply restart from code. In a MySQL primary-replica setup, each replica maintains a copy of data it received from the primary. If the replica pod is replaced with a fresh pod (clean disk), it has no data — it must rebuild from scratch, which can take hours for a large dataset. The pod's disk is its core state.

• REST API servers (Go, Node.js, Python)
• Web front-ends (Next.js, Nuxt)
• Microservices that read from and write to a database
• Background job workers (Celery, Sidekiq, BullMQ)
• ML inference servers (Triton, TorchServe)
• Reverse proxies and load balancers (nginx, HAProxy)

All of these can run as Deployments.

• Relational databases: PostgreSQL, MySQL, MariaDB
• Document databases: MongoDB, CouchDB
• Message queues: Apache Kafka, RabbitMQ (with durable queues)
• Search engines: Elasticsearch, OpenSearch
• Key-value stores (clustered): Redis Cluster, Redis Sentinel
• Coordination services: ZooKeeper, etcd (user-managed)
• Time-series databases: InfluxDB, Prometheus (with remote storage)

All of these run as StatefulSets in Kubernetes.

When you apply a Deployment manifest, the Deployment Controller insidekube-controller-manager creates a ReplicaSet. The ReplicaSet Controller then creates the requested number of pod objects in etcd. The Scheduler assigns each pod to a node. kubelet on each node starts the container.

The Deployment Controller runs a continuous reconciliation loop: desired state (replicas: 3) vs actual state (running pods). If actual < desired, it creates pods. If actual > desired, it deletes pods.

kubectl apply → API Server stores Deployment
Deployment Controller → creates ReplicaSet (desired: 3 pods)
ReplicaSet Controller → creates 3 Pod objects in etcd (all Pending)
Scheduler → assigns each pod to a node
kubelet → starts container on each node
Pods become Running

A Deployment creates exactly two types of resources: one ReplicaSet and N pods. On each rolling update, a new ReplicaSet is created (for the new version) and the old one is scaled to 0. Old ReplicaSets are kept for rollback history — by default, the last 10 are retained (controlled by revisionHistoryLimit).

kubectl get all -l app=my-api

# NAME                        READY   STATUS
# pod/my-api-7d4b9c-xk2pq     1/1     Running
# pod/my-api-7d4b9c-ab3df     1/1     Running
#
# NAME                         DESIRED  CURRENT  READY
# replicaset/my-api-7d4b9c     2        2        2
# replicaset/my-api-6c3a8b     0        0        0   ← old RS, kept for rollback

A Deployment never directly creates or manages pods — it delegates that to ReplicaSets. On every update (image change, env var change, etc.), the Deployment Controller creates a new ReplicaSetfor the updated spec, then scales it up while scaling the old ReplicaSet down. This is how rolling updates work: two ReplicaSets exist simultaneously during the transition, and the old one is preserved at 0 replicas for rollback.

The ReplicaSet Controller constantly watches the number of running pods matching its selector. If a pod crashes or is deleted, the controller detects the count dropped below the desired number and immediately creates a new pod — with a new random name and a fresh start. The new pod has no relationship to the old one. Deployment treats all pods as fungible.

Rolling updates are controlled by two fields: maxSurge (how many extra pods above desired count) and maxUnavailable (how many pods can be down at once). The default is 25% for both.

# Example: 4 replicas, maxSurge: 1, maxUnavailable: 1
# Update triggered (new image)

Step 1: Create 1 new pod (surge) → total: 5 pods
Step 2: New pod passes readiness probe → Ready
Step 3: Delete 1 old pod → total: 4 pods
Step 4: Repeat until all 4 pods are on new version

# Commands:
kubectl set image deployment/my-api api=my-api:v2 -n production
kubectl rollout status deployment/my-api -n production

Kubernetes keeps the previous ReplicaSet at 0 replicas. A rollback simply scales the old ReplicaSet back up and scales the current one down. It is fast because the old pods just need to restart — the image is usually already cached on nodes.

# Rollback to previous version
kubectl rollout undo deployment/my-api -n production

# Rollback to a specific revision
kubectl rollout history deployment/my-api -n production
kubectl rollout undo deployment/my-api --to-revision=2 -n production

The ReplicaSet Controller watches the cluster for its pods. When a pod crashes (exits with any code), or when a node fails and pods become Unknown, the controller sees the actual count has dropped below the desired count and creates a replacement pod. The replacement has a new random name, a new IP, and starts fresh. For a stateless application, this is exactly correct behavior.

You can scale manually or automatically. Manual scaling changes spec.replicas in the Deployment. Automatic scaling via HPA (Horizontal Pod Autoscaler) watches CPU, memory, or custom metrics and adjusts the replica count on your behalf. When scaling up, all new pods start simultaneously. When scaling down, the newest pods die first.

# Manual scale
kubectl scale deployment/my-api --replicas=10 -n production

# Auto scale (HPA)
kubectl autoscale deployment/my-api --min=2 --max=20 --cpu-percent=70 -n production

Yes. You can add a PVC to a Deployment spec under volumes. However, all pods in the Deployment share the same PVC — they all mount the same volume. This works for read-only shared data (config files, static assets) but not for databases, where each instance must have exclusive access to its own disk. Mounting one PVC across multiple pods with ReadWriteOnce access is also impossible — the volume can only be mounted on one node at a time.

• Your application does not write data to local disk between requests
• Any pod can handle any request (pods are interchangeable)
• You need fast horizontal scaling (all pods start simultaneously)
• You need easy rolling updates and rollbacks
• You need HPA for automatic scaling

Use it for: web servers, REST APIs, workers, microservices, ML inference.

• Your application writes data to disk that must survive pod restarts
• Each pod instance must have its own dedicated storage
• Pods must start in a specific order (primary before replicas)
• Other pods need to reach a specific pod by a stable hostname
• You are running a database, message queue, or search engine

StatefulSet uses a dedicated StatefulSet Controller inside kube-controller-manager— it does not use a ReplicaSet. The controller manages pods directly by ordinal number. It knows the identity of every pod it owns and will only create a replacement with the same identity (same name, same PVC reference). It enforces sequential ordering: pod N is never created until pod N-1 is Running AND Ready.

# StatefulSet controller reconciliation loop:
Desired: mysql-0, mysql-1, mysql-2 (all Running+Ready)
Actual:  mysql-0 (Running+Ready), mysql-1 (Missing), mysql-2 (Running+Ready)

Gap detected: mysql-1 is missing
Action: create pod named mysql-1 (not mysql-3, not a random name)
        with PVC data-mysql-1 (existing PVC reattached if present)
Wait:   mysql-1 must reach Running+Ready before any other action
Loop:   check again

Deployment assumes all pods are anonymous. This breaks databases because:

• A crashed pod comes back with a new name → replication source hostname changes → replication breaks
• All pods share storage (or none) → no per-replica data isolation
• All pods start simultaneously → replica starts before primary is ready → connection fails

StatefulSet was introduced in Kubernetes 1.5 (stable 1.9) to give pods the identity, storage, and ordering guarantees that databases need but Deployment cannot provide.

StatefulSet creates pods one at a time, in ordinal order, starting from 0. Each pod must reach Running phase AND pass its readiness probe (Ready=True) before the next pod is created. If a pod fails to become Ready, the StatefulSet is blocked — no further pods are created until the blocked pod recovers.

# Scale from 0 to 3 replicas:
t=0s:  Create mysql-0 + PVC data-mysql-0
       Waiting for mysql-0 to be Running+Ready...
t=45s: mysql-0 is Ready
       Create mysql-1 + PVC data-mysql-1
       Waiting for mysql-1 to be Running+Ready...
t=90s: mysql-1 is Ready
       Create mysql-2 + PVC data-mysql-2
       Waiting for mysql-2 to be Running+Ready...
t=130s: mysql-2 is Ready. StatefulSet is fully up.

Pod names follow the pattern: <statefulset-name>-<ordinal>. A StatefulSet named mysql with 3 replicas creates pods namedmysql-0, mysql-1, and mysql-2. This name is permanent — it does not change if the pod crashes, restarts, or is rescheduled to a different node. The ordinal is always a zero-based integer.

Stable Pod identity is the guarantee that a pod's name, DNS hostname, and storage binding remain the same across its entire lifetime — including crashes, restarts, and node changes. For mysql-1: its name is always mysql-1, its DNS is alwaysmysql-1.mysql-headless.default.svc.cluster.local, and its storage is alwaysdata-mysql-1. Nothing changes when the pod restarts.

Databases configure replication by pointing to a specific hostname. MySQL replica is configured: “replicate from mysql-0.mysql-headless”. If that hostname changes (as it would with a Deployment), the replica loses its source and replication breaks. Kafka brokers register themselves with ZooKeeper or KRaft by their hostname — a changed hostname looks like a brand-new broker, not a restarted one, and partition ownership becomes inconsistent. In distributed systems, identity is how nodes find each other and maintain cluster state.

The ordinal is the integer suffix in a pod name: 0 for mysql-0, 1 for mysql-1, etc. The ordinal controls three things:

• Creation order: pods are created from ordinal 0 upward, one at a time
• Deletion order: pods are deleted from the highest ordinal downward
• Storage naming: PVC name = <template-name>-<pod-name>, so ordinal determines which PVC each pod gets

Init containers can extract the ordinal from the pod hostname using:
[[ $(hostname) =~ -([0-9]+)$ ]] && ordinal=$${BASH_REMATCH[1]}
and use it to decide whether to configure the pod as primary (ordinal 0) or replica (ordinal > 0).

StatefulSet uses volumeClaimTemplates in its spec — a template that describes a PVC. When the controller creates pod N, it first checks whether a PVC named<template-name>-<statefulset-name>-N already exists. If it does, the PVC is reattached. If it does not, a new PVC is created from the template. PVCs are never automatically deleted when a pod is removed.

volumeClaimTemplates is a list inside the StatefulSet spec that defines the PVC template each pod should receive. It looks exactly like a PVC definition — storage class, access mode, size — but instead of creating one PVC, Kubernetes creates one PVC per pod using this template as the blueprint.

volumeClaimTemplates:
- metadata:
    name: data          # PVC names: data-mysql-0, data-mysql-1, data-mysql-2
  spec:
    accessModes: ["ReadWriteOnce"]   # one node at a time — required for databases
    storageClassName: gp3-encrypted  # your cloud storage class
    resources:
      requests:
        storage: 100Gi

When the StatefulSet controller is about to create pod N, it checks whether PVCdata-mysql-N already exists. If not, it creates it using the volumeClaimTemplatesspec. The cloud provider's CSI driver (EBS CSI, GCE PD CSI, Azure Disk CSI) then provisions an actual disk and creates a PersistentVolume that binds to the PVC. The pod starts only after the PVC is Bound.

The StatefulSet controller detects the pod is gone (or not Ready). It creates a new pod with the same name(same ordinal). The new pod references the same PVC by name — so the same disk is reattached. If the pod was on a different node, the cloud volume detaches from the old node and reattaches to the new one (this takes 30–90 seconds for cloud storage). Once attached, the database process starts and resumes from exactly where it left off — no data loss, no reconfiguration needed.

StatefulSet rolling updates go in reverse ordinal order: highest ordinal first. For a 3-replica MySQL: mysql-2 is updated first, then mysql-1, then mysql-0 (the primary, last). Each pod must be Running AND Ready before the next one is updated. This way, replicas are validated on the new version before the primary is touched.

# Update image
kubectl set image statefulset/mysql mysql=mysql:8.1 -n production

# Watch the update (reverse ordinal):
# mysql-2 → deleted → recreated with mysql:8.1 → must be Ready
# mysql-1 → deleted → recreated with mysql:8.1 → must be Ready
# mysql-0 → deleted → recreated with mysql:8.1 → must be Ready

kubectl rollout status statefulset/mysql -n production

Scaling up adds pods in ordinal order. If you scale from 3 to 5, mysql-3 is created and must reach Ready before mysql-4 is created. PVCs are created for each new pod. Scaling down removes pods in reverse ordinal order (highest first), but keeps their PVCs.

# Scale up to 5
kubectl scale statefulset mysql --replicas=5 -n production
# Creates: mysql-3 (waits for Ready), then mysql-4

# Scale down to 2
kubectl scale statefulset mysql --replicas=2 -n production
# Deletes: mysql-4 (waits for termination), then mysql-3, then mysql-2
# Keeps:   PVCs data-mysql-2, data-mysql-3, data-mysql-4

Scale-down deletes pods from the highest ordinal downward, one at a time. Each pod must fully terminate before the next one is deleted. PVCs are never deletedduring scale-down — they are retained so data can be recovered if you scale back up. The pod at ordinal 0 is always the last to be removed.

• Your application writes data to local disk that must survive pod restarts
• Each instance needs its own dedicated storage (not shared with other instances)
• Other pods need to address a specific pod by a stable hostname
• Pods must start in a specific order (e.g., primary before replicas)
• Pod identity must survive restarts (Kafka broker ID, Elasticsearch node name, etc.)

Use it for: PostgreSQL, MySQL, MongoDB, Kafka, Elasticsearch, Redis Cluster, ZooKeeper.

• Your application is stateless — StatefulSet adds sequential startup overhead for no benefit
• You do not need per-pod storage — you are paying for N PVCs unnecessarily
• You need fast, parallel scale-out — StatefulSet's sequential startup slows scaling
• You are running a simple dev environment — a Deployment is simpler and faster

Using StatefulSet for a stateless app is like renting an apartment when you need a hotel — more expensive, more complex, and none of the benefits apply.

Deployment. It is designed exactly for this. Pods start in parallel (fast scale-out), updates use maxSurge/maxUnavailable (configurable speed vs availability tradeoff), and HPA integrates seamlessly. No storage overhead, no headless service, no sequential startup delays.

StatefulSet. If your application writes to disk and that data must survive restarts, or if each instance needs its own storage, or if pods must start in order — use StatefulSet. For production databases, consider adding a Kubernetes Operator on top of the StatefulSet to handle automated failover, backups, and replica management.

No. There are three things a Deployment fundamentally cannot do that StatefulSet can:

• Give each pod a permanent name that survives restarts
• Give each pod its own dedicated PVC (via volumeClaimTemplates)
• Guarantee ordered sequential startup (pod N-1 must be Ready before pod N starts)

You can work around some of these with custom controllers or operators, but then you have re-implemented StatefulSet. Use the right tool.

Technically yes, but it is the wrong choice for stateless apps. StatefulSet starts pods sequentially (much slower scale-out), creates PVCs you do not need (wasted cost), and requires a headless service (unnecessary complexity). You also lose the clean rolling update / rollback mechanism of Deployment. StatefulSet adds overhead that provides zero benefit for stateless applications.

• Sequential startup is slow: 10 pods × 30 seconds each = 5 minutes to start. A Deployment starts all 10 simultaneously.
• PVCs cost money: every pod gets a dedicated disk. For stateless apps this is pure waste.
• Headless service is required: extra configuration that stateless apps do not need.
• Complexity: more moving parts means more to debug and maintain.

APIs and web servers are stateless by design. Every request is independent. Any pod can handle any request. Deployment gives you fast parallel startup (critical for traffic spikes), simple HPA integration, clean rolling updates with maxSurge/maxUnavailable, and no storage or headless service overhead. It is the simplest tool that solves the problem correctly.

Databases need all three things StatefulSet provides: stable identity (so replicas always know where the primary is), per-pod storage (so each replica keeps its own data copy), and ordered startup (so the primary is healthy before replicas try to connect). A Deployment gives none of these — and running a database without them causes data loss, replication failures, or both.

Yes. You add a PVC to a Deployment pod template under spec.volumes, referencing an existing PVC by name. All pods in the Deployment share the same PVC — they all read and write to the same disk. This works for shared read-only data (config files, ML model weights) but is unsuitable for databases.

# Deployment with a shared PVC (all pods mount the same volume)
spec:
  template:
    spec:
      volumes:
      - name: shared-data
        persistentVolumeClaim:
          claimName: shared-pvc   # ONE PVC shared by all pods

# All 3 pods mount the SAME disk. Fine for read-only.
# For databases: two pods writing to the same disk = corruption.

Deployment gives you one shared PVC for all pods. Every pod reads and writes the same disk. StatefulSet gives you one dedicated PVC per pod. Each pod has exclusive access to its own disk.

For a database, shared storage is not an option. MySQL pod A and MySQL pod B cannot both write to the same data directory — they would corrupt each other's files immediately. Each replica needs its own disk containing its own copy of the data. Only StatefulSet volumeClaimTemplatesprovides this — Deployment has no equivalent.

If the Deployment uses a shared PVC, the new pod mounts the same PVC — nothing is lost on the shared disk. If the Deployment uses no PVC (only emptyDir or container filesystem), all in-container data is gone. The new pod starts fresh. For a stateless application this is fine — there is nothing valuable to lose locally.

The StatefulSet controller creates a new pod with the same name (e.g., mysql-1). The new pod binds to the existing PVC (data-mysql-1) — the same disk that the old pod was using. All data the old pod wrote to that disk is still there. The database process picks up exactly where it left off. If the pod moved to a different node, the cloud volume detaches from the old node and reattaches to the new one (usually 30–90 seconds).

No — at least not with ReadWriteOnce (RWO) access mode, which is the correct mode for databases. RWO means only one node can mount the volume at a time. If two pods on different nodes tried to mount the same RWO PVC, the second one would fail to start. Even with ReadWriteMany (RWX), sharing a PVC between database pods causes data corruption — two database processes writing to the same directory simultaneously destroys the data structures.

PostgreSQL uses a data directory (/var/lib/postgresql/data) with a specific file format — WAL files, heap files, index files, and control files. The PostgreSQL process holds file locks on these files. If two PostgreSQL processes mount the same data directory simultaneously, they both try to lock the same files and both try to write WAL logs. The result is immediate data corruption — the files become inconsistent and the database cannot recover without a full restore. This is not a Kubernetes limitation; it is fundamental to how PostgreSQL (and all databases) work.

Pod identity is the combination of properties that uniquely and permanently identify a pod: its name (mysql-1), its DNS hostname(mysql-1.mysql-headless.default.svc.cluster.local), and its storage binding(data-mysql-1). In a StatefulSet, all three survive pod restarts. In a Deployment, none of them survive — every restart creates a new random name, new IP, clean disk.

Distributed systems maintain cluster state by tracking node identities. When a MySQL replica restarts, the primary needs to know it is the same node resuming — not a new node joining. Kafka brokers register their broker ID (derived from their hostname) with the cluster metadata. Elasticsearch tracks shard assignments by node name. If the node identity changes on restart, the cluster cannot correlate the returning node with its stored state — it must treat it as a new member, which triggers expensive rebalancing or leaves the cluster in an inconsistent state.

Stable DNS means a pod's DNS hostname never changes, even if the pod crashes, restarts, or moves to a different node. When a StatefulSet pod restarts on a new IP address (10.244.2.50 instead of 10.244.1.30), its DNS record is updated automatically. Other pods that reference it by hostname (mysql-1.mysql-headless) continue to resolve to the new IP without any reconfiguration.

StatefulSet requires a headless service. When Kubernetes creates a pod in a StatefulSet, it registers a DNS A record for the pod in the format:<pod-name>.<service-name>.<namespace>.svc.cluster.local. This DNS record is updated whenever the pod's IP changes. The hostname stays constant — only the IP behind it changes.

# DNS records created by StatefulSet (mysql) + headless service (mysql-headless):
mysql-0.mysql-headless.default.svc.cluster.local → 10.244.1.5
mysql-1.mysql-headless.default.svc.cluster.local → 10.244.2.3
mysql-2.mysql-headless.default.svc.cluster.local → 10.244.3.8

# mysql-1 crashes, restarts on new IP:
mysql-1.mysql-headless.default.svc.cluster.local → 10.244.2.99  (updated automatically)
# Other pods still use the same hostname → they resolve the new IP transparently

A Headless Service is a Kubernetes Service with clusterIP: None. A regular Service creates a virtual IP (ClusterIP) that load-balances traffic across all matching pods. A Headless Service creates no virtual IP — instead, DNS queries for the service return the list of pod IPs directly, and individual pod DNS records (pod-name.service-name.namespace.svc.cluster.local) are created for each pod.

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
  namespace: production
spec:
  clusterIP: None        # This is what makes it headless
  selector:
    app: mysql
  ports:
  - port: 3306

StatefulSet pods must be reachable by their individual hostname (e.g., mysql-1.mysql-headless) so that the primary can replicate to a specific replica, and so that replicas can connect to the primary by a stable address. A regular ClusterIP service load-balances across all pods — you cannot target a specific pod. A headless service creates per-pod DNS records that resolve directly to individual pod IPs, enabling pod-to-pod communication by stable hostname.

The StatefulSet spec has a required serviceName field that must match the headless service name. Without it, per-pod DNS records are not created.

Deployment pods are anonymous — no other component needs to address a specific pod by name. A Deployment's client sends a request to the Service IP, and the Service load-balances to any pod. The client does not care which pod handles the request. This is the correct design for stateless apps. Per-pod DNS records are unnecessary overhead for an anonymous, interchangeable pod set.

Pods in a StatefulSet communicate via the headless service DNS. A replica connecting to the primary uses: mysql-0.mysql-headless.default.svc.cluster.local:3306. When mysql-0 restarts with a new IP, the DNS record updates automatically — the replica reconnects to the same hostname and gets the new IP. No manual reconfiguration is needed.

Deployment scaling changes spec.replicas. The ReplicaSet Controller reconciles immediately: when scaling up, it creates all needed pods simultaneously; when scaling down, it deletes from newest to oldest. HPA can automate this based on CPU, memory, or custom metrics. Scale-out is fast because pods are created in parallel.

StatefulSet scaling is sequential by default (podManagementPolicy: OrderedReady). When scaling from 3 to 5: mysql-3 is created first and must be Running AND Ready beforemysql-4 is created. Each new pod also gets a new PVC created. This makes StatefulSet scale-out significantly slower than Deployment.

Ordered creation is essential for databases with primary-replica relationships. The primary (mysql-0) must be fully initialized and accepting connections before the first replica (mysql-1) starts — because the replica immediately connects to the primary to begin data synchronization. If the replica starts first, it tries to connect to a non-existent or still-initializing primary, which causes errors or a broken replication state.

Reverse-order deletion during scale-down ensures replicas stop before the primary. If the primary (mysql-0) were stopped first, replicas would lose their replication source mid-operation and enter an error state. Stopping from highest to lowest ordinal means: all replicas stop first, then the primary stops last — the cluster drains cleanly.

Deployment pods are stateless and have no dependencies between them. Pod A does not need Pod B to be ready before it can start. They are all equivalent. Parallel startup is simply faster — if you need 10 pods and they all start in parallel, you are ready in 30 seconds. Sequential startup would take 5 minutes. For stateless apps, there is no reason to be sequential.

The ReplicaSet Controller uses a heuristic: when scaling down, kill the newest pod first. The reasoning: older pods have warmer caches, more established connections, and have proven they can survive under load. A pod running for 2 hours is more stable than one running for 3 minutes. When capacity must be reduced, the least-proven pod is the best choice to remove. This is not random — it is age-based, newest first.

Deployment creates a new ReplicaSet for the updated pod spec, then gradually scales the new RS up while scaling the old RS down. The maxSurge field controls how many extra pods can exist during the transition, and maxUnavailable controls how many pods can be down. New and old pods run simultaneously during the rollout. The rollout is complete when the old RS is at 0.

StatefulSet updates pods in reverse ordinal order, one at a time. The highest-ordinal pod is deleted and recreated with the new spec first, then the next, down to ordinal 0. Each pod must be Running AND Ready before the next is updated. The primary (ordinal 0) is the last pod to be updated.

Using partition: N in the rolling update strategy lets you update only pods with ordinal ≥ N, enabling canary releases on a single replica before rolling out to all.

For databases, it is critical to validate the new version on a replica before updating the primary. If the new image has a bug and mysql-2 crashes after update, mysql-0 (the primary) is still on the old, stable version — you can roll back without any data risk. If the primary were updated first and it crashed, the entire cluster would lose its write leader. Sequential reverse-ordinal updates are a risk-management strategy.

Yes. kubectl rollout undo statefulset/<name> reverts the pod spec to the previous version. The rollback applies the previous spec in reverse ordinal order — exactly like a forward update. Unlike Deployment, there is no previous ReplicaSet to scale up — Kubernetes re-applies the old spec by updating pods in order.

# Rollback StatefulSet
kubectl rollout undo statefulset/mysql -n production

# Check history
kubectl rollout history statefulset/mysql -n production

# Rollback to specific revision
kubectl rollout undo statefulset/mysql --to-revision=2 -n production

Deployment rollback: fast. The old ReplicaSet still exists at 0 replicas with the old pod spec. Rollback scales the old RS back up and the current RS down. Old pods restart from a cached image.

StatefulSet rollback: slower. There is no separate RS to switch back to. Kubernetes re-applies the old spec by updating pods one at a time in reverse ordinal order — the same sequential process as a forward update. Expect full rollout time, not instant RS swap.

The ReplicaSet Controller detects the pod is gone (actual count dropped below desired). It creates a new pod immediately — with a new random name, a new IP, and a clean start. Any in-container data from the crashed pod is gone. The new pod picks up from code, not from any local state. For a stateless application, this is exactly correct — the pod restores from external state (database, cache).

The StatefulSet controller detects the pod is missing. It creates a new pod with the same ordinal name(e.g., mysql-1). The new pod references the existing PVC (data-mysql-1), which is reattached. The database process starts, reads its data from the PVC, and resumes. Other pods in the cluster continue using the same DNS hostname — they transparently reconnect to the new pod IP when the old one goes offline.

The node goes NotReady. After ~40 seconds, pod status becomes Unknown. After the node.kubernetes.io/unreachable toleration expires (default ~5 minutes), the pod is evicted. The ReplicaSet Controller creates a new pod on a healthy node. Because Deployment pods are stateless, the new pod is fully functional immediately. Total recovery time: ~5–6 minutes.

Same eviction sequence (node NotReady → pod Unknown → eviction after ~5 min). After eviction, the StatefulSet controller creates mysql-1 on a healthy node. The cloud volume (data-mysql-1) must detach from the failed node and reattach to the new node — this takes 30–90 seconds for most cloud providers. Total recovery: ~6–10 minutes.

If the storage is hostPath (local to the node), the pod is rescheduled back to the same nodeby Kubernetes topology constraints. If the node is permanently dead, the pod stays Pending forever. This is why local storage is not safe for production StatefulSet databases.

If the PVC backing a StatefulSet pod is deleted while the pod is running, Kubernetes blocks the deletion (a finalizer is added to the PVC). The PVC stays in Terminating state until the pod is stopped. When the pod is eventually deleted and recreated, the StatefulSet controller creates a new empty PVCwith the same name. The new pod starts with a blank disk. All previous data is lost. If the underlying PV had reclaimPolicy: Retain, the PV can be manually rebound.

Pods are deleted in reverse ordinal order: mysql-2 → mysql-1 → mysql-0. Each pod waits for full termination before the next is deleted.PVCs are kept by default — they are not deleted when the StatefulSet is deleted. You must manually delete PVCs if you want to free the storage. In Kubernetes 1.27+, you can configure persistentVolumeClaimRetentionPolicy.whenDeleted: Deleteto auto-delete PVCs.

The Deployment controller deletes the ReplicaSet, which deletes all pods. If the Deployment used a shared PVC, that PVC is not deleted — PVC lifecycle is independent of the Deployment. You must manually delete the PVC if you want to remove the storage. Deleting a Deployment is safe for stateless apps — nothing valuable is lost.

A single-instance PostgreSQL can technically run as a Deployment with a PVC. But this is unsafe for two reasons: (1) if the pod is evicted and rescheduled to a different node, the PVC must reattach — this works with cloud storage but takes time and leaves a window of data unavailability; (2) if you ever try to add a replica (primary + standby), each replica must have its own disk — which requires StatefulSet. For anything beyond a single throwaway dev instance, use StatefulSet.

PostgreSQL in production runs as a primary with one or more streaming replicas. Each replica connects to the primary using a specific hostname and continuously streams WAL logs to stay in sync. If the primary pod restarts with a new name, replicas lose their source and replication breaks. Each replica also stores its own copy of the data — per-pod PVCs are required. StatefulSet provides both: stable primary hostname and dedicated per-pod storage.

Same reasons as PostgreSQL. In a MySQL primary-replica setup, replicas are configured with:CHANGE MASTER TO MASTER_HOST='mysql-0.mysql-headless'. This hostname must be stable. If the primary pod restarts as a Deployment pod with a new name, the MASTER_HOST is invalid and replication fails. StatefulSet ensuresmysql-0 is always mysql-0, regardless of how many times it restarts.

MongoDB Replica Set members discover each other and vote for a primary using stable hostnames. Each member is initialized with: rs.add("mongo-1.mongo-headless:27017"). If a member pod restarts with a new hostname (as with a Deployment), the replica set treats it as a new unknown member and does not grant it voting rights. This can leave the replica set without quorum and unable to elect a primary. StatefulSet preserves hostnames across restarts.

Each Kafka broker has a broker ID — a unique integer that is stored on disk in the meta.properties file. Partition leadership and replica assignment are tracked by broker ID. If a broker restarts and gets a different hostname (Deployment behavior), it reads a different broker ID from disk — or finds no disk at all (if PVC is not attached) — and appears as a new broker. The cluster must rebalance partition assignments, which causes minutes of degraded performance or data inconsistency. StatefulSet ensures the broker always gets the same hostname and the same disk.

Database replication is configured by pointing one server at another by hostname or IP. The replica maintains a persistent connection to the primary and continuously streams changes. If the primary's hostname or IP changes (as happens when a Deployment pod restarts), the replica loses its connection and cannot reconnect — it does not know the new address. In clustered databases (MongoDB Replica Set, Kafka, Elasticsearch), nodes track each other by hostname in persistent configuration stored in etcd or on disk. A changed hostname breaks cluster membership tracking and can cause split-brain or loss of quorum.

Distributed databases run consensus algorithms (Raft, Paxos, Zab) that require a majority of known nodes to agree. Each node in the quorum is identified by its hostname. If node identities change, the quorum loses track of which nodes are which — it cannot distinguish a returning node (should be accepted with its existing data) from a brand-new node (should start from scratch). A wrong identity means the cluster may reject a legitimate node, lose quorum, and stop accepting writes entirely. Stable networking is what makes the cluster self-healing rather than requiring manual intervention on every restart.

Use a Kubernetes Operator (CloudNativePG, Crunchy Data PGO, Zalando postgres-operator) rather than a raw StatefulSet. Operators add automated failover, streaming replica management, backup scheduling, and connection pooling on top of the StatefulSet primitives. If using a raw StatefulSet: 3 replicas, cloud-backed PVCs (gp3, pd-ssd), pod anti-affinity across nodes and zones, PodDisruptionBudget (minAvailable: 2), Guaranteed QoS (requests == limits), and a dedicated write service pointing to the primary.

# production-mysql-statefulset.yaml  (all best practices in one file)
apiVersion: v1
kind: Service
metadata:
  name: mysql-headless          # headless — per-pod DNS records
  namespace: production
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
  - port: 3306
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-write             # ClusterIP — points to primary only
  namespace: production
spec:
  selector:
    app: mysql
    role: primary               # label set by init container on mysql-0
  ports:
  - port: 3306
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: mysql-pdb
  namespace: production
spec:
  minAvailable: 2               # always keep 2 of 3 replicas up during drain
  selector:
    matchLabels:
      app: mysql
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: production
spec:
  serviceName: mysql-headless   # must match headless service name
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: mysql
            topologyKey: kubernetes.io/hostname   # no two replicas on same node
      containers:
      - name: mysql
        image: mysql:8.0
        resources:
          requests:
            cpu: "2"
            memory: 8Gi
          limits:
            cpu: "2"            # requests == limits → Guaranteed QoS (evicted last)
            memory: 8Gi
        readinessProbe:
          exec:
            command: ["mysqladmin", "ping", "-h", "127.0.0.1"]
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data                # PVCs: data-mysql-0, data-mysql-1, data-mysql-2
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3-encrypted    # cloud-backed, encrypted, fast
      resources:
        requests:
          storage: 100Gi

Yes, in production — even for a single instance. A StatefulSet with replicas: 1gives you stable pod identity and PVC reattachment if the pod moves to a different node. A Deployment with a PVC technically works too, but you lose the guarantee that the same PVC is reattached on reschedule (Deployment does not control PVC binding). For development or CI, Deployment is fine — it is simpler and faster to set up.

Yes, for local development or ephemeral CI environments where you do not care about data persistence. A Deployment is simpler: no headless service, no volumeClaimTemplates, faster startup. Just use an emptyDir volume — the database starts fresh every time the pod restarts, which is fine for tests. Never use this pattern in staging or production.

• Running as Deployment: pod crashes → new name → PVC not reattached → data loss.
• Using hostPath or emptyDir: pod moves to new node → data gone.
• No anti-affinity: two replicas on same node → node failure = total data loss.
• No PodDisruptionBudget: node drain during upgrade evicts all DB pods simultaneously → cluster goes down.
• requests ≠ limits: Burstable QoS → evicted before your API servers under node memory pressure.
• No Operator: raw StatefulSet requires manual failover when primary crashes — could take minutes in production.

• Use cloud-backed StorageClass (EBS gp3, GCE pd-ssd, Azure managed-csi) — not hostPath
• Set requests == limits for memory on database containers (Guaranteed QoS)
• Add podAntiAffinity across nodes and cloud zones
• Create a PodDisruptionBudget: minAvailable: N/2 + 1 (quorum)
• Use a Headless Service for DNS + a ClusterIP service for writes + a ClusterIP service for reads
• Use an Operator for Day-2 operations (failover, backup, user management)
• Test PVC expansion (resize) and restore (from backup) before you need it in production

• Always set a readiness probe — without it, pods receive traffic before the app is ready
• Set resource requests and limits on all containers
• Add a preStop: exec: sleep 5 hook to let iptables rules propagate before SIGTERM
• Handle SIGTERM gracefully — drain in-flight requests before exiting
• Set revisionHistoryLimit to keep the last 3–5 ReplicaSets for rollback
• Use HPA with custom metrics (request rate, queue depth) rather than just CPU

Kubernetes does not decide this automatically. Nothing in the StatefulSet or Kubernetes API knows what a “primary” or “replica” is. You teach it to the pods using the ordinal number embedded in the pod hostname.

The convention — used by every PostgreSQL, MySQL, and MongoDB StatefulSet in production — is:

• Ordinal 0 (postgres-0) = primary. It is created first, initializes a fresh data directory, and starts accepting writes.
• Ordinal 1, 2, N (postgres-1, postgres-2) = replicas. They start after the primary is Ready, copy data from the primary, and begin streaming replication.

This logic runs inside an init container — a container that runs to completion before the main database container starts. The init container reads the pod hostname, extracts the ordinal, and branches: if ordinal is 0, configure as primary; otherwise, configure as replica.

For ordinal 0, the init container initializes a brand-new PostgreSQL data directory using initdband writes the configuration needed for replication: wal_level = replica,max_wal_senders = 10, and the replication user entry in pg_hba.conf.

# init container script — runs BEFORE the postgres container starts
#!/bin/bash
set -e

# Extract ordinal from hostname. postgres-0 → ordinal=0, postgres-2 → ordinal=2
[[ $(hostname) =~ -([0-9]+)$ ]] && ordinal=${BASH_REMATCH[1]}

PGDATA=/var/lib/postgresql/data

if [[ ${ordinal} -eq 0 ]]; then
  echo "==> ordinal 0: configuring as PRIMARY"

  # Only initialize if data directory is empty (first start)
  if [ -z "$(ls -A ${PGDATA})" ]; then
    gosu postgres initdb -D ${PGDATA}

    # Enable WAL streaming for replicas
    cat >> ${PGDATA}/postgresql.conf << EOF
wal_level = replica
max_wal_senders = 10
max_replication_slots = 10
hot_standby = on
synchronous_commit = local
EOF

    # Allow replicas to connect for replication
    echo "host replication replicator 0.0.0.0/0 md5" >> ${PGDATA}/pg_hba.conf
  fi

  echo "==> PRIMARY config done. Starting postgres..."
fi

For ordinal > 0, the init container runs pg_basebackup — a PostgreSQL utility that copies the entire data directory from the primary over the network. This gives the replica a consistent starting snapshot. After the copy, the init container writes a standby.signal file (PostgreSQL 12+) and a primary_conninfo setting that tells the replica exactly where to stream WAL logs from: postgres-0.postgres-headless — the stable DNS hostname of the primary.

# init container script — continued for replicas (ordinal > 0)
else
  echo "==> ordinal ${ordinal}: configuring as REPLICA"

  # Only run base backup if data directory is empty
  if [ -z "$(ls -A ${PGDATA})" ]; then
    PRIMARY_HOST="postgres-0.postgres-headless.default.svc.cluster.local"

    echo "==> Waiting for primary at ${PRIMARY_HOST}..."
    # Retry until primary is ready (it might still be initializing)
    until pg_isready -h ${PRIMARY_HOST} -U postgres; do
      sleep 3
    done

    echo "==> Running pg_basebackup from primary..."
    pg_basebackup       -h ${PRIMARY_HOST}       -U replicator       -D ${PGDATA}       --wal-method=stream       --write-recovery-conf       -P

    # PostgreSQL 12+: signal file tells postgres to start in standby mode
    touch ${PGDATA}/standby.signal

    # Tell replica WHERE to stream WAL from (the primary's stable DNS)
    cat >> ${PGDATA}/postgresql.conf << EOF
primary_conninfo = 'host=${PRIMARY_HOST} port=5432 user=replicator application_name=$(hostname)'
recovery_target_timeline = 'latest'
EOF
  fi

  echo "==> REPLICA config done. Starting postgres..."
fi

No — not with a raw StatefulSet. When postgres-0 crashes, Kubernetes restarts it on the same or a new node and reattaches the same PVC (data-postgres-0). postgres-0 comes back as primary again because its init container sees ordinal 0 and finds an existing (non-empty) data directory, so it skips re-initialization and postgres starts normally.

postgres-1 and postgres-2 stay as replicas the whole time — they just pause streaming during the downtime and reconnect when postgres-0 comes back. There is no automatic promotion. This means: if postgres-0's disk is permanently destroyed, your cluster has no primary and the application cannot write. That window of unavailability is why production setups use an operator with leader election.

Patroni is a Python-based high-availability agent for PostgreSQL. Every pod runs Patroni as the main process (not postgres directly — Patroni manages the postgres process). Patroni uses a distributed consensus store (etcd, ZooKeeper, or Kubernetes API itself) to runleader election.

• On startup, all Patroni instances race to acquire a distributed lock. The winner becomes primary and starts postgres in read-write mode.
• Losers become replicas and stream from the elected primary.
• If the primary's Patroni loses the lock (pod crash, network partition), remaining Patroni instances immediately hold a new election. The winner promotes itself to primary in seconds — no human intervention.
• Ordinal 0 is NOT always the primary with Patroni. Any pod can be elected primary based on which one wins the lock.

# How Patroni handles primary/replica — no ordinal dependency
postgres-0  →  Patroni running  →  won the etcd lock  →  PRIMARY (postgres in R/W)
postgres-1  →  Patroni running  →  lost the lock       →  replica (streaming from postgres-0)
postgres-2  →  Patroni running  →  lost the lock       →  replica (streaming from postgres-0)

postgres-0 crashes:
  postgres-1 Patroni: "lock is free, I'll take it" → promotes to PRIMARY (~10s)
  postgres-2 Patroni: lost the race                → replica (now streams from postgres-1)
  postgres-0 restarts: sees postgres-1 holds lock  → rejoins as replica

# In production: use CloudNativePG or Zalando postgres-operator
# Both use Patroni under the hood and add backup, monitoring, user management

Your app connects via the ClusterIP services — not the headless service. The headless service is for pod-to-pod DNS inside the StatefulSet itself (replication config). Your app should use two separate services: one for writes (points only to the primary) and one for reads (load-balances across all pods).

# Three services your app talks to:

# 1. Writes — only reaches the primary (labeled role: primary)
POSTGRES_WRITE_HOST=postgres-write.production.svc.cluster.local

# 2. Reads — load-balances across primary + all replicas
POSTGRES_READ_HOST=postgres-read.production.svc.cluster.local

# 3. Headless — DO NOT use from your app
#    postgres-headless is for StatefulSet pods to find each other
#    DNS returns individual pod IPs, not a stable load-balanced address

PostgreSQL can handle a limited number of simultaneous connections (default: 100). If you have 50 API pods each opening 5 connections, that is 250 connections — your database is at capacity and new requests fail. Connection pooling puts a proxy (PgBouncer) between your app and the database. Apps connect to PgBouncer (thousands of connections allowed), and PgBouncer maintains a small pool (10–30 connections) to PostgreSQL — reusing idle connections instead of opening new ones.

WITHOUT pooling:
50 API pods × 5 connections = 250 connections → PostgreSQL at limit, errors

WITH PgBouncer:
50 API pods × 5 connections → PgBouncer (250 client connections)
                            → PostgreSQL (20 server connections, reused)

PgBouncer sits as a separate Deployment in your cluster:
  App → pgbouncer-service:5432 → postgres-write:5432 (PostgreSQL primary)

podManagementPolicy controls how StatefulSet creates and deletes pods during scale operations. There are two values:

• OrderedReady (default): pods are created one at a time in ordinal order (0→1→2), each waiting for Ready before the next starts. Deleted in reverse (2→1→0).
• Parallel: all pods are created and deleted simultaneously, like a Deployment. No ordering, no waiting for Ready between pods.

This setting only affects scale operations. Rolling updates always go in reverse ordinal order regardless of this setting.

# OrderedReady (default) — for databases with primary/replica hierarchy
apiVersion: apps/v1
kind: StatefulSet
spec:
  podManagementPolicy: OrderedReady   # default, can be omitted
  # mysql-0 starts → Ready → mysql-1 starts → Ready → mysql-2 starts

---
# Parallel — for peer databases where all nodes are equal
apiVersion: apps/v1
kind: StatefulSet
spec:
  podManagementPolicy: Parallel
  # All pods start simultaneously — faster scale-out

Use Parallel when your StatefulSet pods are peers with no hierarchy — no pod needs to wait for another to be ready before it starts. In these systems, each pod independently discovers the cluster and joins it. Sequential startup adds unnecessary delay without any correctness benefit.

• Apache Cassandra: every node is equal, any node can handle any request. Ring topology — nodes discover each other via seed nodes, not by waiting for ordinal 0.
• Elasticsearch: initial cluster formation can happen in parallel. Nodes elect a master among themselves via leader election, not by ordinal.
• Scylla DB: same as Cassandra (compatible protocol).