ClusterIP vs NodePort vs LoadBalancer: Kubernetes Services Explained in Depth

1. Why Service Types Matter

Kubernetes Services are not just an abstraction layer. They are the contract between your application's consumers and the ephemeral, constantly-rescheduled pods that actually do the work. Pods come and go. Their IP addresses change every time they restart. The only stable identity your application has in Kubernetes is its Service — and the type of that Service determines how traffic reaches it, from where, at what cost, and with what guarantees.

Get the Service type wrong and you get exactly what happened in that incident: a fragile, manually-managed connection that collapses the moment Kubernetes does what it was designed to do (reschedule workloads automatically).

There are five Service types in Kubernetes: ClusterIP, NodePort,LoadBalancer, ExternalName, and Headless (technically ClusterIP with clusterIP: None). Each solves a different problem. Understanding when to use each — and more importantly, when not to — separates engineers who operate clusters reliably from engineers who fight fires.

2. How Services Work at the Kernel Level

Before we go type by type, you need a mental model of what actually happens when a packet is sent to a Kubernetes Service. Because nothing happens in the Kubernetes control plane at packet time — it's all pre-programmed kernel rules.

The Virtual IP Problem

A ClusterIP Service has an IP address like 10.96.45.20. No network interface on any node or pod has that IP. It exists only as a rule in the kernel's netfilter tables. When a packet is destined for that IP, the kernel intercepts it and rewrites the destination to the real pod IP — before the routing decision is made. This is DNAT: Destination Network Address Translation.

kube-proxy's Role

kube-proxy runs as a DaemonSet on every node. It watches the Kubernetes API for Service and EndpointSlice changes. When you create a Service, kube-proxy translates it into iptables rules (default mode) or IPVS rules (optional high-performance mode) on every node in the cluster. It does not proxy traffic. Its name is misleading: in iptables mode, kube-proxy just programs the kernel and gets out of the way. All actual packet forwarding is done by the Linux kernel.

iptables Mode (Default)

kube-proxy creates chains in the nat table:

• KUBE-SERVICES — entry point, matches on destination ClusterIP
• KUBE-SVC-<hash> — one chain per Service, handles load balancing using probability rules
• KUBE-SEP-<hash> — one chain per endpoint (pod), performs the actual DNAT

Load balancing in iptables mode is probabilistic: each rule has a probability that decrements as you go down the chain. For three pods: 0.33, 0.50, and 1.0 (catch-all). The math gives equal distribution across all three. There is no true round-robin, no awareness of pod load, and no connection draining. It's random.

IPVS Mode (High Performance)

If you set kube-proxy to IPVS mode, it uses the kernel's IP Virtual Server module instead of iptables chains. IPVS supports multiple scheduling algorithms (round-robin, least-connection, source-hash), has O(1) lookup for large Services (iptables is O(n) for rule matching), and handles Services with thousands of endpoints without performance degradation. For clusters with more than a few hundred Services or endpoints, IPVS mode is strongly recommended.

ASCII Diagram: Client to Pod

  Client Pod (10.244.0.3)
       │
       │  dest: 10.96.45.20:80  (ClusterIP — no real NIC has this IP)
       ▼
  ┌──────────────────────────────────────────────────┐
  │  Linux Kernel netfilter — PREROUTING / OUTPUT    │
  │                                                  │
  │  iptables nat KUBE-SERVICES                      │
  │    match: -d 10.96.45.20 -p tcp --dport 80       │
  │    jump:  KUBE-SVC-XXXXXXXX                      │
  │                                                  │
  │  iptables nat KUBE-SVC-XXXXXXXX                  │
  │    33% prob → KUBE-SEP-AAA (Pod-0: 10.244.1.5)   │
  │    50% prob → KUBE-SEP-BBB (Pod-1: 10.244.2.7)   │
  │    100% →     KUBE-SEP-CCC (Pod-2: 10.244.3.9)   │
  │                                                  │
  │  KUBE-SEP-AAA:                                   │
  │    DNAT → 10.244.1.5:8080                        │
  └────────────────────────┬─────────────────────────┘
                           │  dest is now a real pod IP
                           ▼
                    Pod-0  10.244.1.5:8080

The conntrack table records this translation so that reply packets are reverse-NATted back to the original ClusterIP without the application ever knowing a rewrite happened.

3. ClusterIP Deep Dive

ClusterIP is the default Service type. When you kubectl apply a Service without specifying a type, you get a ClusterIP. It assigns a virtual IP from the cluster's service CIDR (typically 10.96.0.0/12by default in kubeadm clusters) and makes the Service reachable only from within the cluster.

What “Virtual IP” Actually Means

The ClusterIP is not assigned to any network interface. It exists only as a match target in iptables/IPVS rules. You cannot ping a ClusterIP from a node using standard tools unless those tools generate TCP/UDP traffic (because ICMP does not match the TCP/UDP iptables rules kube-proxy installs). This confuses engineers who try to diagnose Service issues with ping and incorrectly conclude the Service is unreachable.

DNS Name

CoreDNS automatically creates a DNS A record for every ClusterIP Service in the format:

<service-name>.<namespace>.svc.cluster.local

A pod in the same namespace can resolve payments-api (short name). A pod in a different namespace must use payments-api.payments or the fully qualified payments-api.payments.svc.cluster.local. The short-name resolution works because the pod's /etc/resolv.conf has search domains configured by the kubelet.

How kube-proxy Programs Rules for ClusterIP

When you create a ClusterIP Service, kube-proxy on every node installs the iptables chains described in section 2. When a pod is added or removed (endpoint changes), kube-proxy updates the KUBE-SVC-* chains to rebalance probabilities. There is a deliberate delay here: kube-proxy watches the API, not the kubelet. An endpoint is removed from the chain only after the pod is removed from the EndpointSlice, which happens after the pod's readiness probe fails or the pod is deleted. During rolling deployments, traffic can still reach terminating pods for a few seconds. This is why you need preStop hooks.

Headless Services: ClusterIP: None

Setting clusterIP: None creates a headless Service. No virtual IP is assigned. Instead, CoreDNS returns the actual IP addresses of the matching pods directly as A records. The caller gets all pod IPs and does its own client-side load balancing or selection. This is how StatefulSets work — you get stable DNS names per pod: redis-0.redis-cluster.cache.svc.cluster.local, redis-1.redis-cluster.cache.svc.cluster.local, etc. Each resolves to the specific pod's IP.

When to Use ClusterIP

• Internal microservice-to-microservice communication
• Database services that should never be exposed outside the cluster
• Any backend that an Ingress controller routes to
• Monitoring endpoints (Prometheus scraping internal services)

Production ClusterIP YAML

# Production ClusterIP Service — annotated
apiVersion: v1
kind: Service
metadata:
  name: payments-api
  namespace: payments
  labels:
    app: payments-api
    env: production
  annotations:
    # Prometheus ServiceMonitor will scrape this port
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    prometheus.io/path: "/metrics"
spec:
  # type: ClusterIP is the default — explicit for clarity
  type: ClusterIP
  selector:
    # Must match Pod labels exactly — this is how Endpoints are built
    app: payments-api
    env: production
  ports:
    - name: http
      port: 80          # Port the Service exposes to other pods
      targetPort: 8080  # Port the container listens on
      protocol: TCP
    - name: metrics
      port: 9090
      targetPort: 9090
      protocol: TCP
  # sessionAffinity: ClientIP  # Uncomment for sticky sessions (5-minute TTL default)
  # sessionAffinityConfig:
  #   clientIP:
  #     timeoutSeconds: 300

4. NodePort Deep Dive

NodePort is a ClusterIP Service with an extra layer: kube-proxy additionally opens a port (in the range 30000–32767) on every node's network interface and routes traffic received on that port to the Service. The port range exists to avoid conflicts with standard OS ports and applications typically running below 1024.

How It Actually Works

When you create a NodePort Service, kube-proxy installs an additional rule in the KUBE-NODEPORTS chain that matches on the assigned NodePort and jumps to the same KUBE-SVC-* chain used by the ClusterIP. Traffic arriving on NodeIP:31000 gets DNAT'd to a pod, which may be on a completely different node. The ClusterIP still exists and works as normal.

externalTrafficPolicy: Cluster vs Local

This is one of the most misunderstood settings in Kubernetes networking.

externalTrafficPolicy: Cluster (default) — Any node that receives traffic on the NodePort will forward it to any pod in the cluster, regardless of whether that pod is local. This means an extra network hop is likely (the packet leaves the receiving node for another node). Source IP is SNATted to the node's IP so the reply can return correctly. The pod sees the node IP, not the client's real IP.

externalTrafficPolicy: Local — Traffic on the NodePort is only forwarded to pods that are running on that specific node. If no local pod exists, the connection is dropped. Source IP is preserved because no SNAT is needed. The tradeoff: uneven load distribution if pods are not spread across all nodes.

Source IP Preservation

If your application needs the real client IP (for rate limiting, geolocation, audit logs), you must useexternalTrafficPolicy: Local with NodePort, or use a LoadBalancer Service with Proxy Protocol. With the default Cluster policy, the source IP your pod receives is the node IP, not the client's real IP. Rate limiting based on source IP will rate-limit nodes, not clients — which is exactly what happened in Incident 2 below.

Security Implications

NodePort opens a port on every node, including master/control-plane nodes in some configurations. If your security group allows traffic on the NodePort range from 0.0.0.0/0, every node in your cluster is directly reachable from the internet on that port. Most production environments restrict NodePort traffic to specific source CIDRs or use it only internally behind a Load Balancer.

When NOT to Use NodePort in Production

• Direct internet exposure — Use a LoadBalancer Service or an Ingress controller instead
• Hardcoded IPs — The incident in the intro. Never. Use DNS or a LoadBalancer hostname.
• When you need source IP — Unless you're also using externalTrafficPolicy: Local
• Multi-port services with conflicting ports — The assigned NodePort is random by default

5. LoadBalancer Deep Dive

A LoadBalancer Service is a NodePort Service with an additional controller loop: the cloud controller manager watches for LoadBalancer-type Services and provisions a real cloud load balancer (AWS NLB/CLB, GCP TCP LB, Azure Load Balancer) that points at the cluster nodes' NodePorts. Kubernetes then updates the Service'sstatus.loadBalancer.ingress field with the external IP or hostname.

Cloud Controller Manager Integration

The cloud controller manager runs as a separate deployment (on managed clusters like EKS, GKE, AKS it's managed for you). It watches Service objects and reconciles them against the cloud provider's API. When you create a LoadBalancer Service, within 30–90 seconds, a real cloud LB appears. When you delete the Service, the LB is deleted too. The mapping is one Service → one cloud LB unless you use shared Ingress controllers.

AWS NLB vs ALB

On AWS, the default LoadBalancer Service type provisions a Classic Load Balancer (CLB, also called ELB v1), which is legacy and should be avoided. Use the annotation service.beta.kubernetes.io/aws-load-balancer-type: "nlb"to get an NLB (Network Load Balancer, L4). For HTTP/HTTPS with advanced routing, use the AWS Load Balancer Controller add-on and create an Ingress resource pointing to a ClusterIP Service — that will provision an ALB (Application Load Balancer, L7). You do not get an ALB from a Service resource directly.

Health Checks and Target Groups

When you use externalTrafficPolicy: Local with a LoadBalancer Service, the NLB health-checks each node's NodePort. If a node has no local pods for that Service, the health check fails and the NLB removes that node from its target group. This is how source IP preservation works at the LB level: the LB only sends traffic to nodes that have a local pod. No SNAT, no extra hop, real client IP visible to the pod.

This only works correctly if the NLB health check interval is short enough. If health check interval is 30 seconds and unhealthy threshold is 3, a node can receive traffic for up to 90 seconds after its last pod is gone. The aws-load-balancer-healthcheck-interval: "10" and healthcheck-unhealthy-threshold: "2"annotations above cut that to 20 seconds — acceptable for most use cases.

Costs

Each LoadBalancer Service provisions a separate cloud LB. In AWS, an NLB costs roughly $16/month in hourly charges plus data transfer. If you have 50 microservices each with their own LoadBalancer Service, you're spending $800+/month just on load balancers. This is the primary reason production clusters use an Ingress controller: one LB (or a small number) handling routing for all services.

LoadBalancer Service YAML with AWS NLB

# LoadBalancer Service with AWS NLB and externalTrafficPolicy: Local
apiVersion: v1
kind: Service
metadata:
  name: api-gateway
  namespace: production
  annotations:
    # Use NLB (Network Load Balancer) not CLB
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    # Internal NLB (remove for internet-facing)
    # service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    # Cross-zone load balancing — keeps costs down, improves distribution
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    # Proxy Protocol v2 — needed to preserve client source IP at L4
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    # Health check config — tune to your app's startup time
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
    # Reuse existing EIP/Elastic IP allocations
    # service.beta.kubernetes.io/aws-load-balancer-eip-allocations: "eipalloc-xxx,eipalloc-yyy"
spec:
  type: LoadBalancer
  # Local = NLB only sends to nodes that have a local pod
  # This preserves source IP and removes the extra hop
  externalTrafficPolicy: Local
  selector:
    app: api-gateway
  ports:
    - name: http
      port: 80
      targetPort: 8080
    - name: https
      port: 443
      targetPort: 8443

Full Traffic Flow Diagram

  ┌─────────────────────────────────────────────────────────────────────────┐
  │        LoadBalancer → Node → kube-proxy → Pod  (full path)              │
  └─────────────────────────────────────────────────────────────────────────┘

  Client 203.0.113.50
        │
        │  TCP :80
        ▼
  ┌──────────────────────────────────┐
  │  AWS NLB  52.10.0.100            │
  │  Listener: TCP:80                │
  │  Target Group: Node:31080        │
  └───────────────┬──────────────────┘
                  │
        ┌─────────┴──────────┐
        │                    │
        ▼                    ▼
  Node-A :31080         Node-B :31080   ← NLB picks a node
  10.0.0.11             10.0.0.12
        │
        │  iptables DNAT (kube-proxy)
        ▼
  KUBE-NODEPORTS → KUBE-SVC-XXX → KUBE-SEP-YYY
        │
        │  DNAT to real pod
        ▼
  Pod-2  10.244.3.7:8080   ← may be on a DIFFERENT node (externalTrafficPolicy:Cluster)

  ─────────────────────────────────────────────────────
  With externalTrafficPolicy: Local

  NLB health-checks each node's NodePort.
  Only nodes WITH a local pod pass the health check.
  NLB only sends traffic to nodes that have a local pod.
  No extra hop → source IP is preserved.
  ─────────────────────────────────────────────────────

6. ExternalName Service

ExternalName is a special Service type that creates a DNS CNAME record instead of assigning a ClusterIP or programming iptables rules. No load balancing, no proxy, no kube-proxy involvement. When a pod resolves the Service name, CoreDNS returns the CNAME, and the client follows it.

apiVersion: v1
kind: Service
metadata:
  name: legacy-database
  namespace: production
spec:
  type: ExternalName
  externalName: rds-prod.us-east-1.rds.amazonaws.com

A pod resolving legacy-database.production.svc.cluster.local gets back a CNAME pointing tords-prod.us-east-1.rds.amazonaws.com. This is useful for:

• Referencing external databases or APIs from within the cluster using internal DNS names
• Migrating services: point the ExternalName at an external service first, then swap to a ClusterIP Service pointing to in-cluster pods once migration is complete, without changing any consumer config
• Multi-cluster setups where a service in one cluster needs to reach a service in another

7. Headless Services

A headless Service is created by setting clusterIP: None. No virtual IP. kube-proxy does not touch it. CoreDNS, on the other hand, handles it differently: it creates individual A records for each pod IP rather than a single A record for a VIP.

DNS Behavior

For a normal ClusterIP Service with three pods, a DNS query returns one IP (the ClusterIP). The kernel handles load balancing via iptables. For a headless Service with three pods, a DNS query returns three IPs (the three pod IPs) directly. The caller is responsible for selecting one.

StatefulSet Integration

StatefulSets require a headless Service for stable pod identity. Each StatefulSet pod gets a DNS record in the format <pod-name>.<service-name>.<namespace>.svc.cluster.local. This is stable across pod restarts — redis-0.redis-cluster.cache.svc.cluster.local always resolves to the pod named redis-0, regardless of what node it's scheduled on or what IP it has.

This is fundamental to distributed systems running in Kubernetes. Redis Cluster, Cassandra, ZooKeeper, Kafka, and Elasticsearch all depend on stable peer addresses. Without a headless Service, you cannot implement these systems correctly in Kubernetes.

Headless Service YAML for StatefulSet

# Headless Service for a StatefulSet (Redis Cluster example)
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster
  namespace: cache
  labels:
    app: redis-cluster
spec:
  # clusterIP: None makes this headless
  # No virtual IP is assigned. DNS returns pod IPs directly.
  clusterIP: None
  selector:
    app: redis-cluster
  ports:
    - name: redis
      port: 6379
      targetPort: 6379
    - name: cluster-bus
      port: 16379
      targetPort: 16379
  # publishNotReadyAddresses: true  # Include not-ready pods in DNS
  # Useful during StatefulSet initialization when pods need to discover
  # each other before becoming Ready
---
# The StatefulSet that references this headless Service
# Each pod gets a stable DNS name:
# redis-cluster-0.redis-cluster.cache.svc.cluster.local
# redis-cluster-1.redis-cluster.cache.svc.cluster.local
# redis-cluster-2.redis-cluster.cache.svc.cluster.local

8. Ingress vs Service — When to Use What

Ingress is not a Service type. It is a separate Kubernetes resource that defines L7 routing rules (host-based, path-based) and is implemented by an Ingress controller (nginx, Traefik, HAProxy, AWS ALB controller, etc.). The Ingress controller itself runs as a Deployment with a LoadBalancer or NodePort Service fronting it.

The main reason to use Ingress over individual LoadBalancer Services is cost and consolidation. One Ingress controller handles all HTTP routing. One cloud LB. One set of TLS certificates to manage. The Services the Ingress routes to are all ClusterIP — they are not exposed externally at all.

9. Service DNS Deep Dive

CoreDNS

CoreDNS runs as a Deployment in the kube-system namespace (typically 2 replicas). It watches Kubernetes API for Services and Endpoints and serves DNS for the cluster.local domain. Every pod is configured to use CoreDNS as its resolver via /etc/resolv.conf, which is written by the kubelet at pod startup.

Search Domains and ndots

The /etc/resolv.conf in a pod looks like:

nameserver 10.96.0.10
search payments.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The ndots:5 setting means: if a hostname has fewer than 5 dots, try appending each search domain before trying it as an absolute name. payments-api has zero dots, so CoreDNS triespayments-api.payments.svc.cluster.local first (which works). But www.example.com has 2 dots, so CoreDNS tries appending search domains first (3 failed lookups) before trying www.example.comas-is. This causes latency on external DNS lookups from pods.

The fix for external lookups: always append a trailing dot to force absolute resolution —www.example.com. — or reduce ndots for your application pods. Some platforms setndots:2 for pods that do mostly external lookups.

DNS Record Types

• ClusterIP Service: A record pointing to the ClusterIP. SRV records for named ports.
• Headless Service: Multiple A records, one per pod IP.
• StatefulSet pod: A record per pod (pod-0.svc.ns.svc.cluster.local).
• ExternalName Service: CNAME to the externalName value.

10. SessionAffinity and EndpointSlices

SessionAffinity

Setting sessionAffinity: ClientIP on a Service tells kube-proxy to send requests from the same client IP to the same pod. It uses iptables recent module or IPVS source-hash scheduling. The default TTL is 10800 seconds (3 hours).

The limitations are significant. “ClientIP” is the source IP as seen by the node. WithexternalTrafficPolicy: Cluster, that is the node IP doing the SNAT, not the real client — so session affinity will route all traffic that passed through one particular node to the same pod, completely defeating the intent. Session affinity also does not handle pod restarts — if a pod dies, conntrack entries expire and new connections are distributed randomly again.

For real session stickiness, use your application layer (JWT, session tokens, application-level routing) or a proper L7 load balancer with cookie-based affinity (an Ingress controller that supports it).

EndpointSlices

Originally, Kubernetes maintained a single Endpoints object per Service containing all pod IPs. At scale (Services with hundreds of endpoints), a single endpoint change triggered a full resync of the entire Endpoints object to every node — O(n) API traffic. EndpointSlices, introduced in Kubernetes 1.17 and GA in 1.21, shard endpoints into slices of up to 100 entries each. An update to one pod only updates one slice, not the entire set. kube-proxy consumes EndpointSlices by default since Kubernetes 1.22.

You should not manage EndpointSlices directly. They are created and maintained by the EndpointSlice controller. For troubleshooting, kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<svc>shows the current pod IPs backing a Service.

Service Types Side-by-Side

  ┌─────────────────────────────────────────────────────────────────────────┐
  │                  KUBERNETES SERVICE TYPES — SIDE BY SIDE                │
  └─────────────────────────────────────────────────────────────────────────┘

  ClusterIP (default)              NodePort                  LoadBalancer
  ─────────────────────            ────────────────────────  ──────────────────────────
  Only reachable from              Reachable from outside    Reachable from internet
  inside the cluster               via NodeIP:NodePort        via cloud LB IP

  [Pod A]                          Internet / VPC            Internet
      │                                │                         │
      │ service.ns.svc.cluster.local   │ 10.0.0.12:31000         │ 203.0.113.5:80
      ▼                                ▼                         ▼
  [ClusterIP: 10.96.x.x:80]       [Node eth0]               [Cloud Load Balancer]
      │                                │                         │
      │ kube-proxy DNAT                │ kube-proxy DNAT          │ target group / NLB
      ▼                                ▼                         ▼
  [Pod B :8080]                   [Pod :8080]               [Node:NodePort]
                                                                 │
                                                                 ▼
                                                            [Pod :8080]

  ExternalName                     Headless (clusterIP: None)
  ────────────────────             ────────────────────────────
  DNS CNAME alias                  No virtual IP — direct DNS

  [Pod]                            [Client]
      │                                │
      │ mydb.svc.cluster.local          │ DNS query: mystatefulset-0.svc
      ▼                                ▼
  [DNS CNAME]                      [CoreDNS returns Pod IPs directly]
      │                                │
      ▼                                ▼
  [rds.us-east-1.amazonaws.com]    [Pod-0 :6379] [Pod-1 :6379] [Pod-2 :6379]

kube-proxy iptables Chain Detail

  ┌─────────────────────────────────────────────────────────────────────────┐
  │           kube-proxy iptables chains for a ClusterIP Service            │
  └─────────────────────────────────────────────────────────────────────────┘

  Packet destined to 10.96.45.20:80 (ClusterIP)

  PREROUTING / OUTPUT (nat table)
        │
        ▼
  KUBE-SERVICES chain
  ──────────────────────────────────────────────────────────
  -d 10.96.45.20/32 -p tcp --dport 80 → KUBE-SVC-XXXXXXXX
        │
        ▼
  KUBE-SVC-XXXXXXXX  (load balancing chain — one rule per endpoint)
  ──────────────────────────────────────────────────────────
  -m statistic --mode random --probability 0.33 → KUBE-SEP-AAA  (Pod-0)
  -m statistic --mode random --probability 0.50 → KUBE-SEP-BBB  (Pod-1)
  (no probability match)                         → KUBE-SEP-CCC  (Pod-2)
        │
        ▼
  KUBE-SEP-AAA  (endpoint chain)
  ──────────────────────────────────────────────────────────
  DNAT → 10.244.1.5:8080   (actual Pod IP:Port)

  Result: virtual ClusterIP 10.96.45.20:80 → real pod 10.244.1.5:8080
  The kernel's conntrack table remembers the translation for session continuity.

11. Three Production Incidents

Incident 1: The Hardcoded NodePort IP (45-Minute Outage)

Incident 2: externalTrafficPolicy:Cluster Breaks Rate Limiting

Incident 3: Headless Service Misconfiguration Causes Split-Brain in Redis Cluster

12. Troubleshooting Playbook

# 1. List all Services in a namespace
kubectl get svc -n payments

# 2. Show full Service spec including ClusterIP, NodePort assignments
kubectl get svc payments-api -n payments -o yaml

# 3. Check Endpoints (legacy) — are pods being selected?
kubectl get endpoints payments-api -n payments
# Empty Endpoints = label selector mismatch or pods not Ready

# 4. Check EndpointSlices (current — replaces Endpoints at scale)
kubectl get endpointslices -n payments -l kubernetes.io/service-name=payments-api

# 5. Describe the Service — events show provisioning errors for LB type
kubectl describe svc payments-api -n payments

# 6. Test ClusterIP reachability from inside the cluster
kubectl run curl-test --image=curlimages/curl:8.6.0 -it --rm --restart=Never -- \
  curl -v http://payments-api.payments.svc.cluster.local/health

# 7. Test DNS resolution directly
kubectl run dns-test --image=busybox:1.36 -it --rm --restart=Never -- \
  nslookup payments-api.payments.svc.cluster.local

# 8. Check kube-proxy iptables rules on a node (requires node access)
# iptables -t nat -L KUBE-SERVICES -n | grep payments
# iptables -t nat -L KUBE-SVC-<hash> -n

# 9. Watch real-time Endpoint changes
kubectl get endpoints payments-api -n payments -w

# 10. Check if LoadBalancer external IP is assigned
kubectl get svc api-gateway -n production \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Systematic Diagnosis Flow

1. Is the Service created? kubectl get svc -n <namespace>
2. Does it have endpoints? kubectl get ep <svc> -n <namespace> — if empty, check Pod labels vs Service selector and Pod readiness
3. Can you reach the ClusterIP from a test pod? kubectl run test --image=curlimages/curl -it --rm -- curl http://<clusterip>:<port>
4. Does DNS resolve? nslookup <svc>.<ns>.svc.cluster.local from inside a pod
5. For LoadBalancer: is external IP assigned? Check EXTERNAL-IP column — if pending, cloud controller manager issue
6. Is the target port correct? targetPort must match the container's listening port, not the Service port
7. NetworkPolicy blocking traffic? kubectl get networkpolicy -n <namespace> and review ingress rules

13. Common Mistakes

1. Hardcoding node IPs for NodePort access — Nodes are ephemeral. Use a LoadBalancer hostname or DNS.
2. Pinging a ClusterIP to test reachability — ClusterIP has no ICMP rules. Use curl or telnet.
3. Using NodePort to expose services directly to the internet — Opens every node to port scanning. Use a LoadBalancer + security groups.
4. Missing readiness probe causing traffic to unready pods — Endpoints are added when pods become Ready. Without a probe, pods are Ready immediately even if the app hasn't finished starting.
5. Wrong targetPort — port: 80 is the Service port; targetPort: 8080 is the container port. Mismatch = connection refused.
6. Label selector mismatch between Service and Pods — The most common cause of empty Endpoints. Double-check with kubectl get pods -l <selector>.
7. Not using externalTrafficPolicy: Local when source IP matters — Breaks IP-based rate limiting, geolocation, and audit logging.
8. Using ClusterIP Service as StatefulSet governing service — StatefulSets require headless Services for pod DNS names.
9. Not setting publishNotReadyAddresses: true for distributed system init — Causes split-brain during initial cluster formation.
10. Creating one LoadBalancer Service per microservice — Unnecessary cost. Use an Ingress controller for HTTP services.
11. Using sessionAffinity: ClientIP with externalTrafficPolicy: Cluster — Source IP is the node IP after SNAT; affinity routes all of a node's traffic to one pod.
12. Not accounting for ndots:5 latency on external DNS — Each external hostname resolves after 3 failed search-domain lookups. Add trailing dots or reduce ndots for pods making external calls.
13. Forgetting to update security groups when NodePort changes — Kubernetes may assign a different port if you delete and recreate a Service without specifying nodePort explicitly.
14. Ignoring LB health check settings — Default AWS health check settings mean a removed pod can still receive traffic for 60-90 seconds. Tune thresholds and intervals.
15. Using ExternalName for TLS termination — ExternalName is DNS-only. TLS is handled end-to-end between client and external service with no in-cluster interception.
16. Not setting preStop hooks on terminating pods — Endpoints are removed from iptables rules with a delay. Without a preStop sleep (typically 5–15 seconds), in-flight connections to a terminating pod fail.

14. Interview Q&A

Beginner Questions

Q1: What is a Kubernetes Service and why do we need it?

What the interviewer is testing: Do you understand the fundamental problem Services solve, or do you just know the definition?

Common wrong answer: “A Service exposes your application.” Too vague.

Ideal answer: Pods are ephemeral. They get new IP addresses every time they restart, reschedule, or scale. A Service provides a stable virtual IP and DNS name that other pods or external clients can reliably address. kube-proxy keeps the Service's backend pod list (Endpoints) synchronized as pods come and go. Without Services, you'd have to constantly update every client with new pod IPs — which is unmanageable at any scale.

Q2: What is the difference between ClusterIP and NodePort?

What the interviewer is testing: Can you explain the access boundary and the mechanism?

Common wrong answer: “ClusterIP is internal, NodePort is external.” Technically correct but missing the mechanism.

Ideal answer: ClusterIP creates a virtual IP accessible only from within the cluster. kube-proxy programs iptables on every node so any pod can reach the virtual IP, which gets DNAT'd to a real pod. NodePort extends this by additionally opening a port (30000–32767) on every node's external network interface, so traffic arriving from outside the cluster on that port gets forwarded to the Service. ClusterIP is internal; NodePort adds an external access path but it's coarse-grained and not suitable as the primary external access mechanism in production.

Q3: What happens if a Service has no matching pods?

What the interviewer is testing: Understanding of Endpoints and readiness.

Common wrong answer: “An error is thrown.”

Ideal answer: The Service exists and has a stable ClusterIP and DNS name, but its Endpoints object is empty (no addresses). kube-proxy installs no KUBE-SEP rules. Connections to the ClusterIP will be dropped by the kernel because there is no DNAT rule to rewrite them. kubectl get endpoints <svc> returns the Service name with no addresses. Fix: correct the Service's selector to match the pod labels, or ensure pods pass their readiness probe.

Q4: What port does a NodePort Service use?

Ideal answer: A port in the range 30000–32767 (configurable via kube-apiserver flag--service-node-port-range). Kubernetes assigns one randomly if not specified. You can request a specific port in the Service spec with the nodePort field, as long as it's within the range and not already in use. The same port is opened on every node in the cluster.

Q5: How does a pod find another Service by name?

Ideal answer: Via DNS. CoreDNS runs in the cluster and watches Services. Each Service gets an A record: <name>.<namespace>.svc.cluster.local. Pods have /etc/resolv.confconfigured with the CoreDNS ClusterIP as nameserver and search domains including svc.cluster.local. So a pod in the same namespace can just use the short Service name and it will be resolved via the search domain expansion to the full FQDN.

Intermediate Questions

Q6: Explain externalTrafficPolicy and when you'd use Local vs Cluster.

What the interviewer is testing: Real operational understanding — have you dealt with source IP issues?

Common wrong answer: Defining the policy without mentioning the tradeoff.

Ideal answer: With Cluster (default), any node receiving traffic will forward it to any pod, potentially via an extra node hop. The node doing the forward SNATs the source IP so the pod sees the node IP, not the client IP. Good for load distribution, bad for source-IP-dependent features. With Local, traffic is only forwarded to pods local to the receiving node. No SNAT, so source IP is preserved. Bad for load distribution if pods are unevenly spread. For LoadBalancer Services with Local policy, the cloud LB health-checks each node and only routes to nodes with local pods, which compensates for the uneven distribution issue. I always use Local with NLBs when source IP matters (rate limiting, geolocation, audit logging).

Q7: What is a headless Service and when would you use one?

Ideal answer: A headless Service has clusterIP: None. No virtual IP is assigned. CoreDNS returns the individual pod IPs directly. Used for StatefulSets where each pod needs a stable, unique DNS name (pod-0.svc.ns.svc.cluster.local). Also used when the client needs to do its own load balancing (gRPC clients that maintain persistent connections, some database drivers, Redis Cluster clients that need direct pod addressing). Key distinction: ClusterIP Services hide individual pod IPs; headless Services expose them.

Q8: How does kube-proxy implement load balancing for a Service?

What the interviewer is testing: Kernel-level understanding.

Ideal answer: In iptables mode, kube-proxy creates a chain per Service with one rule per endpoint. Each rule has a probability that decrements proportionally: for N pods, first rule matches at probability 1/N, second at 1/(N-1), etc. This gives even random distribution. There is no round-robin, no health-aware balancing, and no connection draining. In IPVS mode, the kernel's Virtual Server subsystem is used, which supports multiple scheduling algorithms (round-robin, least-connection, source-hash) and performs at O(1) rather than iptables' O(n) rule matching. For clusters with many Services or many endpoints per Service, IPVS is significantly more efficient.

Q9: What causes a Service to have empty Endpoints?

Ideal answer: Three main causes. First, label selector mismatch: the Service selector doesn't match any pod labels — verify with kubectl get pods -l <selector-labels> -n <namespace>. Second, pods are not Ready: if pods are running but failing readiness probes, they are excluded from Endpoints — check kubectl describe pod for readiness probe failures. Third, namespace mismatch: the Service is in a different namespace from the pods (Services only select pods in their own namespace). All three are diagnosable in under 2 minutes with the right kubectl commands.

Q10: What is an EndpointSlice and why was it introduced?

Ideal answer: The original Endpoints object stored all pod IPs for a Service in a single object. At scale (hundreds of pods per Service), any pod change triggered a full resync of the entire object to every node — O(n) API and network overhead. EndpointSlices shard endpoints into slices of up to 100 entries. A pod IP change only updates one slice. This dramatically reduces API server load and network traffic in large clusters. EndpointSlices also carry more metadata (topology hints for zone-aware routing, conditions likeserving/terminating per endpoint). kube-proxy consumes EndpointSlices by default since Kubernetes 1.22.

Advanced Questions

Q11: How would you design the service networking for a multi-tenant cluster where tenants must not reach each other's Services?

What the interviewer is testing: Systems design thinking with Service, DNS, and NetworkPolicy.

Ideal answer: Services in different namespaces are reachable by name with the full FQDN. Isolation requires NetworkPolicy. Each tenant namespace gets a default-deny ingress NetworkPolicy, then allow-list rules permitting only traffic from their own namespace (or specific labeled namespaces like monitoring). ClusterIP Services are still technically reachable if the destination pod allows it — NetworkPolicy enforces at the pod level, not the Service level. For stronger isolation, consider using a CNI with L7 policy (Cilium) or a service mesh with mTLS (Istio, Linkerd) so that connections are authenticated at the application layer, not just filtered by IP. The headless Service vector is important to consider: headless Services expose pod IPs, not just a VIP, so NetworkPolicy coverage must be complete.

Q12: A LoadBalancer Service has been pending for 10 minutes with no external IP assigned. Walk me through your diagnosis.

What the interviewer is testing: Systematic troubleshooting of the cloud controller manager loop.

Ideal answer: First, kubectl describe svc <name> — look at Events. If there are events from the cloud controller manager, they'll describe the failure (insufficient IAM permissions, subnet not tagged, VPC issue). Second, check cloud controller manager logs:kubectl logs -n kube-system -l component=cloud-controller-manager. Third, verify the cluster has a cloud provider integration at all — a kubeadm cluster without MetalLB or a cloud provider plugin will never fulfill LoadBalancer requests. Fourth, check IAM: on EKS, the node IAM role needselasticloadbalancing:* permissions (or the AWS LBC service account with IRSA if using the AWS Load Balancer Controller). Fifth, check subnet tags: EKS requires subnets tagged with kubernetes.io/role/elb: 1for public LBs or kubernetes.io/role/internal-elb: 1 for internal ones.

Q13: How does Kubernetes handle graceful termination for a Service endpoint?

What the interviewer is testing: Deep understanding of the pod termination / Service convergence race condition.

Ideal answer: When a pod is deleted, multiple things happen in parallel: the kubelet sends SIGTERM to containers, and the Endpoints controller removes the pod from EndpointSlices. kube-proxy on each node watches EndpointSlices and updates iptables to stop routing new connections to the terminating pod. The problem is propagation delay: there can be a 1–5 second window where kube-proxy has not yet received the Endpoints update but the pod is already terminating. New connections arriving during this window hit a terminating pod. The correct mitigation is a preStop hook with a sleep (typically 5–15 seconds) before the container receives SIGTERM — this gives kube-proxy time to drain new connections. Additionally,terminationGracePeriodSeconds must be longer than the preStop sleep plus the app's shutdown time. This is a very common source of in-flight request errors during rolling deployments.

Q14: What are the implications of using IPVS mode vs iptables mode for kube-proxy in a large cluster?

Ideal answer: iptables rule matching is O(n) — each incoming packet must traverse all Service rules until a match is found. With 10,000 Services, that can be tens of thousands of rule traversals per packet. IPVS uses a hash table with O(1) lookup regardless of Service count. For connection-heavy workloads (microservices with high RPS), iptables mode adds measurable latency at scale. IPVS also supports better scheduling algorithms (least-connection, source-hash) vs iptables' only option of random-with-probability. Tradeoffs: IPVS requires the kernel's ip_vs modules loaded (most cloud node images include them) and uses a separate kernel subsystem (ipvsadm for debugging vs iptables -L). The mode is set in the kube-proxy ConfigMap and requires a rolling restart of kube-proxy DaemonSet pods to take effect.

Q15: Explain how Topology-Aware Routing (formerly Topology Hints) works with Services and why you'd use it.

What the interviewer is testing: Awareness of advanced Service features and multi-AZ cost optimization.

Ideal answer: In a multi-AZ cluster, the default Service routing sends traffic to any pod regardless of availability zone. Cross-AZ traffic in AWS costs $0.01/GB. For high-throughput internal Services, this adds up quickly. Topology-Aware Routing (enabled with service.kubernetes.io/topology-mode: Autoannotation) tells the EndpointSlice controller to add topology hints to each endpoint indicating which zone it prefers. kube-proxy on nodes in zone A will prefer endpoints with zone A hints. If no local endpoints exist, it falls back to cross-zone. This requires sufficient endpoint distribution across zones — the controller does not enable hints for very small Services (fewer than 3 endpoints per zone) to avoid uneven load distribution. I've used this to reduce inter-AZ costs by 60-70% on internal data-intensive Services.

15. Best Practices

1. Always name Service ports (e.g., name: http, name: grpc). Named ports allow Ingress controllers, Prometheus, and Istio to identify protocols automatically.
2. Use externalTrafficPolicy: Local for any Service where source IP matters. Pair with health check tuning on the cloud LB.
3. Set publishNotReadyAddresses: true on headless Services for distributed systems that need peer discovery during initialization.
4. Always add a preStop sleep (5–15 seconds) to containers to handle the Endpoints propagation delay during rolling updates.
5. Use ClusterIP Services as the backend for Ingress, not NodePort or LoadBalancer Services. The Ingress controller handles external access.
6. Prefer Ingress over multiple LoadBalancer Services for HTTP/HTTPS workloads. One LB, many routes, fraction of the cost.
7. Use IPVS mode for kube-proxy if you have more than 500 Services or frequently see >10,000 active connections per node.
8. Label your Services consistently with app, env, and version. It makes troubleshooting, canary deployments, and monitoring significantly easier.
9. Always specify targetPort by name if the pod's port name matches — this makes port changes in pods transparent to the Service definition.
10. For LoadBalancer Services on AWS, always specify aws-load-balancer-type: nlb to avoid CLBs (legacy, less capable, being phased out).
11. Set explicit health check intervals and thresholds on NLBs. Don't rely on defaults, which can leave stale endpoints in rotation for up to 90 seconds.
12. Use NetworkPolicy alongside Services for defense in depth. A Service alone does not prevent unauthorized pod-to-pod communication.
13. Avoid sessionAffinity: ClientIP for external services where externalTrafficPolicy: Cluster is set. The source IP will be the node, making affinity ineffective.
14. For StatefulSets, always create the headless Service before the StatefulSet. Kubernetes requires the governing Service to exist before pods are created.
15. Monitor EndpointSlice churn. High churn rate (many endpoint updates per minute) signals instability in pod scheduling or readiness probes. Alert on this metric from kube-state-metrics.
16. Use ExternalName Services as a migration tool when moving services from external to in-cluster: zero client reconfiguration needed.
17. Avoid specifying static NodePorts unless you have a specific reason. Let Kubernetes assign them to avoid conflicts as the cluster grows.
18. Set resource requests/limits on CoreDNS pods proportional to cluster size. Underprovisioned CoreDNS is a common source of mysterious DNS timeout errors in large clusters.
19. Enable service.kubernetes.io/topology-mode: Auto for high-throughput internal Services in multi-AZ clusters to reduce cross-AZ data transfer costs.
20. Document all Services with annotations. At minimum: owner team, on-call runbook URL, and whether the Service is internal or external-facing.

16. Frequently Asked Questions

Can two Services share the same selector?

Yes. Multiple Services can select the same set of pods. This is useful for exposing the same workload on different ports or with different access policies (e.g., an internal ClusterIP Service and an external LoadBalancer Service for the same deployment).

Can I change a Service type after creation?

Yes, with caveats. You can patch or apply a changed type. Kubernetes will remove the old cloud LB and provision a new one if you change from ClusterIP to LoadBalancer. The new LB will have a different IP/hostname, so any DNS records or configs pointing to the old LB must be updated. Note that changing from LoadBalancer to ClusterIP will delete the cloud LB immediately.

Why does my Service have a CLUSTER-IP but I get “connection refused”?

ClusterIP exists but Endpoints are empty (no pods selected), or the pod is listening on a different port thantargetPort specifies, or the pod's container is crashing. Check:kubectl get ep <svc>, kubectl describe pod, and verify the container's actual listening port with kubectl exec <pod> -- ss -tlnp.

Does a Service need to be in the same namespace as the pods it selects?

Yes. A Service's selector only matches pods in its own namespace. Cross-namespace pod selection is not supported. For cross-namespace access, a pod in namespace A calls the Service DNS name in namespace B:svc-name.namespace-b.svc.cluster.local.

What happens to existing connections when a pod is removed from a Service?

Existing connections (tracked in the conntrack table) continue to work until they close or the conntrack entry expires. New connections stop being routed to the removed pod immediately after kube-proxy updates iptables. This is why in-flight request errors can occur during rolling updates: new requests may hit the iptables rule that was just removed, getting dropped, while in-progress requests on established connections continue briefly.

Can I use a Service to load-balance UDP traffic?

Yes. Services support TCP, UDP, and SCTP. For UDP, be aware that kube-proxy's iptables rules work fine, but session affinity using ClientIP is based on source IP/port for UDP — behavior may differ from TCP. For UDP load balancers on cloud providers, check whether your cloud LB type supports UDP (NLBs do, ALBs do not).

What is the difference between port and targetPort in a Service?

port is the port exposed on the Service (the port other pods connect to). targetPort is the port on the pod's container that receives the traffic. They can be different. Typical pattern: Service port 80, container port 8080. You can also use named ports: set the container port name in the Pod spec and reference the name in targetPort — this makes the Service decoupled from the actual container port number.

How do I expose a Service without creating a cloud load balancer?

Options: NodePort (use with VPN or private network access), MetalLB (on-premises bare-metal), Tailscale Operator, or kubectl port-forward (dev only). For on-premises production clusters, MetalLB is the standard solution for LoadBalancer Services.

What does “EXTERNAL-IP: <pending>” mean?

The cloud controller manager has not yet provisioned the cloud load balancer. This is either because it's still in progress (wait 1–2 minutes), the cluster has no cloud integration, IAM permissions are insufficient, or subnet/VPC configuration is incorrect. Check cloud controller manager logs.

Can I have a Service without a selector?

Yes. A Service without a selector creates no Endpoints automatically. You can manually create an Endpoints (or EndpointSlice) object pointing to any IPs — including addresses outside the cluster. Useful for routing Kubernetes service traffic to external databases or legacy systems while keeping the Service DNS name stable.

What is the relationship between an Ingress and a Service?

An Ingress references Services by name. The Ingress controller resolves those Service names to their ClusterIPs or EndpointSlices (depending on the controller) and routes L7 traffic accordingly. The Services referenced in an Ingress are typically ClusterIP type — they don't need to be LoadBalancer type because the Ingress controller handles external access.

How does Kubernetes handle IPv6 or dual-stack Services?

Kubernetes has supported dual-stack (IPv4+IPv6) Services since 1.21 (GA). Set ipFamilyPolicy: PreferDualStackor RequireDualStack and ipFamilies: [IPv4, IPv6]. The cluster must be configured with both IPv4 and IPv6 service CIDRs. Not all CNI plugins support dual-stack; verify compatibility before enabling.

Can I specify which NodePort number a Service uses?

Yes. In the Service spec under ports, set nodePort: 31234 (must be in the configured range, default 30000–32767). If you don't specify it, Kubernetes assigns a random available port in the range. Specifying it explicitly is useful when external load balancers or firewall rules reference fixed NodePorts.

What happens to a Service if CoreDNS is down?

Existing connections and applications that cached DNS records continue working. New DNS lookups fail. Applications that do not cache DNS (resolving on every request) start seeing DNS timeouts. ClusterIP-based iptables rules continue functioning — traffic routing does not depend on CoreDNS being up. But the inability to resolve names typically causes cascading failures in microservice architectures. CoreDNS is a critical dependency and should run with at least 2 replicas and proper resource limits.

Should I use Kubernetes Services or a service mesh for internal service communication?

Both. A service mesh (Istio, Linkerd) operates on top of Kubernetes Services, not instead of them. Services still provide discovery and stable virtual IPs. The mesh sidecar intercepts traffic and adds mTLS, observability, retry logic, and circuit breaking. You do not choose between them; you use Services as the addressing layer and optionally add a mesh for the control plane features.

17. Comparison Tables

Service Types Comparison

externalTrafficPolicy: Cluster vs Local

NetworkPolicy for a ClusterIP Service

# NetworkPolicy scoped to the ClusterIP Service's selector
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-api-ingress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payments-api
      env: production
  policyTypes:
    - Ingress
  ingress:
    # Allow traffic from api-gateway pods in the api-gateway namespace
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: api-gateway
          podSelector:
            matchLabels:
              app: api-gateway
      ports:
        - protocol: TCP
          port: 8080
    # Allow Prometheus scraping from monitoring namespace
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app: prometheus
      ports:
        - protocol: TCP
          port: 9090

18. Key Takeaways

• A Kubernetes Service is a stable network identity (virtual IP + DNS name) that abstracts ephemeral pods. It is the fundamental building block of all Kubernetes networking.
• ClusterIP is for internal communication. It is the correct type for every microservice that is exposed externally only via an Ingress controller.
• NodePort opens a port on every node and is not a suitable primary external access mechanism in production. It is the mechanism under a LoadBalancer Service, not something you use directly for internet traffic.
• LoadBalancer provisions a cloud LB. One per Service = real money. Use Ingress for HTTP services to share one LB across many routes.
• externalTrafficPolicy: Local is required any time you need real client IP in your pods. The default Cluster policy SNATs all external traffic to node IPs.
• Headless Services are non-negotiable for StatefulSets. They provide the pod-stable DNS names that distributed systems like Redis, Cassandra, and Kafka require.
• kube-proxy does not proxy traffic. It programs iptables (or IPVS) rules that the Linux kernel uses to DNAT packets. In iptables mode, all load balancing is probabilistic and O(n). Use IPVS mode at scale.
• DNS is the correct way to address Services. Never hardcode ClusterIPs or node IPs — they change. The DNS name does not.
• Empty Endpoints is always caused by a label selector mismatch or failing readiness probes. Check these two things first, always.
• A preStop hook with a sleep is required for zero-downtime rolling deployments. The Endpoints propagation delay creates a window where terminating pods receive new traffic.