Kubernetes Security Best Practices: RBAC, Secrets, PodSecurity, and the Defaults That Will Get You Breached

Three Months Later. A Different Kind of War Room.

No security alert this time. Fluorescent lights. A conference room at a company you want to work for. The interviewer — calm, unhurried — asks one question:

You take a breath. You survived an incident. You know this. You start listing controls.

The Answer That Gets 80% of Candidates Eliminated

Most people answer with a checklist. Four boxes. Clean. Sounds complete.

The interviewer nods. Writes something down. Then looks up.

Five questions. Most candidates have given the checklist answer so many times they never thought past it. Engineers who can answer all five — and explain the counterintuitive edges — walk out with the offer. Let's answer every one.

Before we go architecture-deep, here's the mental model that makes everything click. Think of your Kubernetes cluster as an office building:

Hold that analogy. Everything below is the exact same thing — except the master badge can be bound with a RoleBinding to limit it to one floor, and the safe isn't actually locked until you configure etcd encryption at rest.

Q1: What Is the Difference Between a Role and a ClusterRole?

Most people say “Role is for one namespace, ClusterRole is for the whole cluster.” That is true but incomplete — and the part they miss is the part that trips up clusters in production.

The correct answer: A Role is namespace-scoped — it grants permissions to resources within a single namespace and can only be bound within that namespace. A ClusterRole is cluster-scoped by definition — it can grant permissions across all namespaces, or for cluster-scoped resources that don't belong to any namespace at all, like Nodes, PersistentVolumes, and StorageClasses.

Here is the counterintuitive part that most tutorials skip entirely: the scope of access is determined by the binding, not by the Role type.

A ClusterRole bound with a ClusterRoleBinding grants cluster-wide access. That is the dangerous one. A ClusterRole bound with a RoleBinding (in namespace X) grants that ClusterRole's permissions — but scoped to that namespace only. This lets you define a reusable role template as a ClusterRole and deploy it narrowly into specific namespaces with a RoleBinding. Many teams do this without knowing it.

  Subject (ServiceAccount, User, Group)
         │
         ▼
  ┌────────────────────────────────────────────┐
  │        RoleBinding (namespace-scoped)       │  ──── binds Role in ONE namespace
  │   or   ClusterRoleBinding (cluster-scoped)  │  ──── binds ClusterRole EVERYWHERE
  └────────────────────────────────────────────┘
         │
         ▼
  ┌────────────────────────────────────────────┐
  │           Role (namespace-scoped)           │  ← namespaced resources only
  │   or   ClusterRole (cluster-scoped)         │  ← can reference any resource type
  └────────────────────────────────────────────┘
         │
         ▼
  ┌────────────────────────────────────────────┐
  │       Kubernetes API Server: ALLOW/DENY     │
  │  (deny-by-default; first ALLOW wins)        │
  └────────────────────────────────────────────┘

  Four combinations and what they mean:
  ─────────────────────────────────────────────────────────────────────
  Role           + RoleBinding         → namespace-scoped (standard)
  ClusterRole    + ClusterRoleBinding  → cluster-wide (dangerous if broad)
  ClusterRole    + RoleBinding         → namespace-scoped (reusable template pattern)
  Role           + ClusterRoleBinding  → not valid (Kubernetes rejects this)
  ─────────────────────────────────────────────────────────────────────

# ── Step 1: A Role is namespace-scoped ──────────────────────────────────────
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payments-config-reader
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["payments-config"]   # lock to a named resource, not all configmaps
    verbs: ["get"]

---
# ── A ClusterRole is cluster-scoped (definition) ────────────────────────────
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
  - apiGroups: [""]
    resources: ["nodes"]       # cluster-scoped resource — only a ClusterRole can grant this
    verbs: ["get", "list", "watch"]

---
# ── Counter-intuitive: ClusterRole + RoleBinding = namespace-scoped access ──
# This binds the ClusterRole but limits it to ONE namespace.
# The binding scope, not the Role type, determines the effective scope.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-sa-rb
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: payments-api-sa
    namespace: payments
roleRef:
  kind: ClusterRole
  name: payments-config-reader   # reusing a ClusterRole, scoped to this namespace only
  apiGroup: rbac.authorization.k8s.io

Q2: Are Kubernetes Secrets Actually Secret?

No.

Not by default. The name “Secret” is marketing. Or historical accident. Either way, it sets an expectation the implementation does not meet.

Kubernetes Secrets are stored in etcd as base64-encoded strings. Base64 is an encoding scheme. It is not encryption. Anyone with kubectl get secret RBAC permission retrieves the value and decodes it in under a second. Anyone with direct access to the etcd backup file reads every secret in the cluster without touching Kubernetes at all.

# What a Kubernetes Secret actually looks like in etcd:
# base64-encoded. Not encrypted. Decodable in 0.3 seconds.
#
# apiVersion: v1
# kind: Secret
# data:
#   password: cGFzc3dvcmQxMjM=   ← echo -n 'password123' | base64

# Decode any secret:
kubectl get secret payments-db-creds -n payments \
  -o jsonpath='{.data.password}' | base64 -d
# → password123
# That's it. That's the "security."

---
# ── The right approach: External Secrets Operator ───────────────────────────
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payments-db-creds
  namespace: payments
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: payments-db-creds
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/database
        property: password
# The Kubernetes Secret is created by ESO, rotated from AWS Secrets Manager,
# audited in AWS CloudTrail. The source of truth is the external store.

The name “Secret” means: “this value is intended to be sensitive, so we store it separately from ConfigMaps and we only expose it to the pods and users with RBAC permission to read it.” That is it. RBAC is the lock. base64 is not.

What actually makes Secrets secure

Three controls, in order of impact:

• etcd encryption at rest. Configure an EncryptionConfiguration on the API server. Use the KMS provider to integrate with AWS KMS, GCP KMS, or Azure Key Vault. Now a stolen etcd backup is useless — the values are encrypted with a key stored outside etcd.
• RBAC with minimal scope. Only the pods that need a secret should have RBAC to read it. Lock down with resourceNames to restrict access to specific named Secrets. get is not the same as list — list secretsreturns all of them.
• External secret stores. AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager. Use the External Secrets Operator to sync values into Kubernetes Secrets automatically. The source of truth lives outside Kubernetes with its own audit trail, rotation policies, and access controls. The Kubernetes Secret becomes a cache. A breach of the cache exposes one rotation cycle, not your credential history.

Q3: What Is a ServiceAccount and Why Does Every Pod Have One by Default?

A ServiceAccount is a Kubernetes identity for a pod. The API server uses it to authenticate calls from within the cluster — to answer: “who is making this API request?” when the caller is a pod, not a human.

Every pod automatically gets the default ServiceAccount for its namespace unless you specify otherwise. That default SA gets a token mounted at/var/run/secrets/kubernetes.io/serviceaccount/token. The token is valid. The token can authenticate to the API server. The API server will then check RBAC to decide what it can do.

Here is the problem: many clusters have — usually via someone following a tutorial — granted the default ServiceAccount broad permissions. Sometimescluster-admin. Sometimes just get secrets on all namespaces. Either way, every pod in that namespace inherits those permissions, including pods deployed by attackers who exploited a vulnerability in your application.

# ── Never use the default ServiceAccount ────────────────────────────────────
# This is what happens by default (a disaster waiting to happen):
#
#   spec:
#     # serviceAccountName: default  ← implicit
#     # automountServiceAccountToken: true  ← implicit
#
# The token is mounted at:
# /var/run/secrets/kubernetes.io/serviceaccount/token
# Any code in the container can read it and call the Kubernetes API.

---
# ── What you should do instead ──────────────────────────────────────────────
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api-sa
  namespace: payments
automountServiceAccountToken: false   # opt-out at the SA level — every pod using this SA
                                       # gets NO token unless explicitly opted in

---
# Then in the Pod spec, only opt-in if the pod genuinely needs API access:
spec:
  serviceAccountName: payments-api-sa
  automountServiceAccountToken: false  # belt and suspenders — explicit

# Check what tokens are being mounted right now:
# kubectl get pods -A -o json | jq \
#   '.items[] | select(.spec.automountServiceAccountToken != false) |
#    .metadata.namespace + "/" + .metadata.name'
# The output will surprise you.

The rule is: explicitly set automountServiceAccountToken: false at the ServiceAccount level unless the pod genuinely needs to call the Kubernetes API. Operators, controllers, and CD tooling need API access. Your stateless API server, your batch job, your frontend pod — they almost certainly do not.

Q4: What Does PodSecurity Admission Actually Prevent?

Most people say “PodSecurity stops privileged containers.” That is correct but it is about a quarter of the answer. The more important question is: what does PodSecurity not prevent, and why does that matter?

PodSecurity Admission (PSA) replaced PodSecurityPolicy in Kubernetes 1.25. It is a built-in admission controller — no installation required — that enforces one of three security profiles at the namespace level via namespace labels.

Three profiles, one clear hierarchy

• Privileged. Anything goes. Use for system namespaces (kube-system, Cilium DaemonSets, Falco). Not for application workloads.
• Baseline. Prevents known privilege escalation vectors. Blocks privileged: true, hostNetwork, hostPID,hostIPC, and hostPath volumes. Good for workloads that cannot immediately meet Restricted.
• Restricted. Hardened. Everything in Baseline, plus: must run as non-root, must set allowPrivilegeEscalation: false, must drop ALL capabilities, and must use seccompProfile: RuntimeDefault or a custom profile. This is what production application workloads should use.

# Namespace labeled for PSA Restricted enforcement:
kubectl label namespace payments \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=v1.30 \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

---
# Pod spec that PASSES the Restricted profile:
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault

  containers:
    - name: payments-api
      image: registry.internal/payments-api:sha256-abc123  # pinned digest, NEVER :latest
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 1000
        capabilities:
          drop: ["ALL"]     # drop every Linux capability — add nothing back
      volumeMounts:
        - name: tmp
          mountPath: /tmp   # only writable path — explicit

  volumes:
    - name: tmp
      emptyDir: {}

# What Restricted BLOCKS:
# - privileged: true
# - hostNetwork / hostPID / hostIPC: true
# - hostPath volumes (any)
# - allowPrivilegeEscalation: true (or absent)
# - running as root (UID 0)
# - capabilities beyond the minimal set

What PSA does NOT prevent

This is the part that matters for the interview:

• Excessive RBAC permissions. PSA enforces what the pod looks like at creation time. It says nothing about what the pod can do once running — that is RBAC's job.
• Insecure images. A Restricted pod can still run an image with 200 HIGH CVEs and a cryptominer baked in. Image scanning and signing are separate controls.
• Lateral movement via the network. A Restricted pod can still make connections to every other pod in the cluster. That is NetworkPolicy's job.
• Secrets already mounted. PSA does not inspect what secrets are in a pod's volumes. A Restricted pod with a mounted cluster-admin ServiceAccount token is still dangerous.

Q5: RBAC Is Read-Only. The Pod Still Reads Secrets. Why?

This is the question that separates engineers who understand the system from engineers who memorised the documentation. The wrong answer: “RBAC controls all secret access.”

The right answer: RBAC controls API access. Volume mounts bypass the API entirely.

When a Secret is mounted as a volume or injected as an environment variable at pod creation time, the kubelet retrieves it from the API server during pod setup and writes it directly to the container's filesystem. Once that happens, the container reads the file directly — no API call, no RBAC check. The file is on the container's local filesystem at whatever mount path you specified.

Limiting the ServiceAccount's RBAC to get secrets prevents the container from making new Kubernetes API calls to read secrets programmatically. It does not evict secrets that were already mounted when the pod started. If your pod spec has a secretKeyRef or a secretVolume, the container will read that secret regardless of what the ServiceAccount's RBAC says.

The implication: your security review cannot stop at “what RBAC does this ServiceAccount have?” You must also review what is mounted in every pod spec. Both vectors matter.

Supply Chain Security: The Attack Vector Nobody Audits Until After

RBAC, NetworkPolicy, and PodSecurity protect your cluster from the inside out. Supply chain security protects you from what enters the cluster in the first place.

The attack surface is: your application code, your dependencies, your base images, your build tools, and your registry. A malicious package introduced via a transitive dependency can deploy a cryptominer just as effectively as an over-privileged ServiceAccount. Both vectors are in the incident reports. Both require controls.

Image scanning

Scan every image with Trivy before it reaches production. Fail CI on HIGH and CRITICAL CVEs with available fixes. Running an image with 47 HIGH vulnerabilities is a choice. It should be a deliberate, documented exception — not the default.

Image signing with Cosign

Cosign signs container images and stores the signature in the OCI registry alongside the image. An admission controller (Kyverno or OPA Gatekeeper) can then block any unsigned image from entering the cluster. An attacker who compromises your registry cannot deploy their image without your CI signing key.

ImagePullPolicy: Always

In production, set imagePullPolicy: Always. The default IfNotPresentmeans a node that already has the image cached will use that cached copy — even if the registry version has been updated to include a security fix. For production workloads, always verify freshness on every start.

Restrict registries via admission

Block Docker Hub and public registries in production via an admission webhook policy. Only images from your internal registry — which you scan, sign, and control — should be deployable. A developer who pulls an unvetted image from Docker Hub in production is a supply chain breach waiting to happen.

# Sign an image with Cosign after CI builds it:
cosign sign --key cosign.key registry.internal/payments-api:v1.2.3

# Verify before deploying:
cosign verify --key cosign.pub registry.internal/payments-api:v1.2.3

---
# Kyverno policy: only allow signed images from your registry
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-image-signature
      match:
        any:
          - resources:
              kinds: ["Pod"]
      verifyImages:
        - imageReferences:
            - "registry.internal/*"
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----
  # If an image is not signed by your CI key, it is blocked at admission.
  # An attacker who compromises the registry cannot deploy their image
  # without the private signing key.

# Kubernetes Audit Policy — the CCTV system for your cluster
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Every Secret access logged with request + response body
  # (you need to know WHAT was returned, not just that it was accessed)
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]

  # Log privilege escalation attempts (new ClusterRoleBindings, etc.)
  - level: Request
    verbs: ["create", "update", "patch"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["clusterrolebindings", "rolebindings", "clusterroles", "roles"]

  # Log exec into pods — the most common post-compromise action
  - level: Request
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # Drop read-only noise from system components
  - level: None
    verbs: ["get", "list", "watch"]
    users: ["system:kube-proxy", "system:serviceaccount:kube-system:kube-proxy"]

  # Default: metadata only for everything else
  - level: Metadata

The Production Disasters (Two Real Failures, One Lesson)

Story 1: The tutorial cluster-admin

A developer joined a six-person startup. They were setting up Helm in a new EKS cluster and hit a permission error on the third helm install. A Stack Overflow answer from 2019 said to run:

kubectl create clusterrolebinding default-admin --clusterrole=cluster-admin --serviceaccount=default:default

It worked. Helm installed. The developer moved on. Nobody documented it. Nobody reviewed it. The cluster grew. Six months later: a third-party analytics container had a server-side request forgery vulnerability. The attacker used it to read the pod's mounted ServiceAccount token. That token had cluster-admin.

The audit logs — which did exist — showed the API calls: kubectl get secrets -Aequivalent, then database credentials exfiltrated, then a cryptominer deployed via the API, then lateral movement to three other namespaces. The ServiceAccount binding had been there the whole time. Audit logs caught it after 72 hours, not in real time, because nobody had set alerts.

Prevention: monthly ClusterRoleBinding audit. Falco alert on unexpectedkubectl exec and bulk secret reads. The cluster-admin binding was visible in plain sight in kubectl get clusterrolebindings for six months. No one looked.

Story 2: Secrets in environment variables in CI logs

A fintech team stored their database password as a Kubernetes Secret and mounted it as an environment variable in their API deployment. Standard practice. Secure enough.

A junior developer, debugging a startup failure, ran kubectl describe podand pasted the output into the team's public Slack channel to ask for help. The output included the full environment variable list, including aDATABASE_URL that was not from a Secret — it was a plain value:field in the Deployment manifest. Someone had added it during a debug session fourteen months earlier and never removed it. The password was in plaintext in Slack. In fourteen months of build logs. In three Git commit histories.

This was not a Kubernetes failure. It was a Secret hygiene failure. The Kubernetes Secret was fine. The plain value: field in the Deployment was not. The lesson: securing the Kubernetes object is not enough. You need to audit the full propagation path: Deployment manifest → Git → CI logs → pod describe output → Slack. At each step, is the value exposed?

The Wall of Shame: Eight Mistakes That Are in Production Right Now

1. cluster-admin on the default ServiceAccount. Giving every visitor to the building a master key to the server room because they needed to use the printer. The default ServiceAccount is shared by every pod in the namespace that doesn't specify otherwise. Granting it cluster-admin grants cluster-admin to every unspecified pod in the cluster. An attacker who gets code execution in any of those pods owns the cluster. This is the exact mechanism of the 2:17 AM incident. Run kubectl get clusterrolebindings -o wideright now. Fix what you find.
2. Kubernetes Secrets are base64, and people think that's encryption. Hiding a spare key under the doormat and calling it a “secure location.” Base64 is reversible by every Unix system in the world with one command. Without etcd encryption at rest, a stolen etcd backup is a complete credential dump. Enable encryption at rest. Use an external secret store. Treat every unencrypted etcd backup as a breach event.
3. No audit logging configured. A building with no CCTV. You know someone took something. You have no idea who, when, or what they touched on the way out. Without audit logs, a breach investigation is guesswork. The audit policy is 40 lines of YAML. The regret of not having it when you need it is considerably longer.
4. Running containers as root. Giving every contractor in the building full admin rights to all systems because it was easier than scoping their access. Running as UID 0 inside a container does not give you root on the host — until there is a container escape vulnerability. At that point, root in the container means root on the node. SetrunAsNonRoot: true and runAsUser: 1000 in the securityContext. Every container. Every workload. No exceptions.
5. imagePullPolicy: IfNotPresent in production. Trusting that the milk you bought last week is still fine because you don't want to check. A cached image on a node is the version that was there when the node was provisioned or when the image was last pulled. If a security patch was released and you updated the registry, nodes with cached images will not pick it up until forced. In production, imagePullPolicy: Always. The extra second on pod start is not worth the compromise window.
6. No NetworkPolicy (flat network). An office where every employee can walk into every room because “we trust our people.” The moment one person is compromised, every room is compromised. A default-deny NetworkPolicy with explicit allow rules means a compromised frontend pod cannot reach the database directly. The blast radius of a compromise is contained to the permitted network paths. Thirty lines of YAML prevent lateral movement across the entire cluster.
7. Wildcard RBAC rules (resources: ["*"], verbs: ["*"]). Writing an access badge that says “yes to everything” because listing specific rooms seemed tedious. A wildcard RBAC rule is cluster-admin by another name. It grants access to resources that don't even exist yet. Every new resource type added to the cluster is automatically accessible to every wildcard binding. Always enumerate specific resources and verbs. Always.
8. Secrets mounted as env vars, then logged in CI. Writing your password on a post-it, photographing it, uploading the photo to twelve different systems, and being surprised when it leaks. Environment variables appear in crash dumps, in kubectl describe pod output, in CI job logs when the environment is printed during debugging, and in application logs when the application does what many applications do and logs its configuration at startup. Mount secrets as files. Read them at the specific path. Never log the environment.

Production Best Practices

1. Audit all ClusterRoleBindings quarterly. Any non-system subject with cluster-admin is a finding. Treat it as one. Fix it in the same sprint you find it.
2. automountServiceAccountToken: false on every ServiceAccount by default. Opt-in per pod for the workloads that genuinely need API access. This is the single control that would have prevented the cryptominer incident.
3. Enable etcd encryption at rest with the KMS provider. A stolen etcd backup should be useless. Without this, it is a complete credential dump.
4. Apply PSA Restricted to every production namespace. Label the namespace. Run in audit mode first to see what breaks. Remediate. Then enforce.
5. Default-deny NetworkPolicy in every namespace on day one. Include the DNS egress exception. Open only the network paths you explicitly need.
6. Use External Secrets Operator with an external store. AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager. The Kubernetes Secret is a cache. The external store is the source of truth.
7. Configure audit logging with RequestResponse on secrets. Alert when unexpected subjects access secrets. Without this, a breach investigation is guesswork.
8. Sign all production images with Cosign. Enforce signing in admission. An attacker who compromises your registry cannot deploy without your CI signing key. Kyverno can block unsigned images at admission.

FAQ

Can I use PodSecurityPolicy instead of PSA?

No. PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed in 1.25. If you are running 1.25 or later (which is likely), PSP does not exist. PodSecurity Admission is the replacement — it is built in, requires no installation, and is enforced via namespace labels. If you are on an older cluster still using PSP, migrating to PSA before upgrading is the correct sequence.

Does Flannel enforce NetworkPolicies?

No. Flannel does not implement NetworkPolicy enforcement. If you create a NetworkPolicy in a Flannel cluster, it is accepted by the API server and silently ignored at the network level. All pods still have unrestricted access to all other pods. NetworkPolicy enforcement requires a CNI plugin that supports it: Calico, Cilium, Antrea, or Weave. Verify your CNI before relying on NetworkPolicies for security.

What is the difference between seccomp and AppArmor?

seccomp restricts which system calls a process can make. AppArmor restricts which file paths a process can access, which network operations it can perform, and which capabilities it can use. They operate at different levels and are complementary. For most clusters, starting with seccompProfile: RuntimeDefault (blocks ~300 unusual syscalls with zero application changes) is the right first step. AppArmor runtime/default is available on Ubuntu nodes and adds file path restrictions on top.

How do I rotate Secrets without downtime?

With External Secrets Operator: set a refreshInterval on the ExternalSecret. When you rotate in the external store, ESO picks up the new value within one refresh interval and updates the Kubernetes Secret. Applications that read secrets from mounted files pick up the new value dynamically — Kubernetes updates the mounted file when the Secret changes (with up to 60 seconds propagation delay). Applications that read from environment variables require a pod restart. This is why mount-as-file beats inject-as-env for secrets that rotate.

Is it ever acceptable to use privileged: true in production?

Rarely. Legitimate cases: node-level observability DaemonSets that need eBPF access (Falco, Cilium, Datadog Agent), some storage CSI drivers, and specific network plugin DaemonSets. These belong in dedicated system namespaces with strong RBAC and explicit PSA Privileged labeling — not in application namespaces. If an application developer says their workload needs privileged: true, the correct response is: “Which specific capability does it need?” It is almost always possible to grant a specific Linux capability instead of full privilege.

🎤 The 60-Second Interview Answer

Back in the interview room. The whiteboard is still there. You've answered all five follow-up questions. Here is how you deliver the complete answer — covering the surface-level checklist and the architecture depth that gets you the offer: