CKS — Kubernetes Secrets Deep Dive

9 minute read

Kubernetes Secrets Deep Dive

CKS Exam Domain: Minimize Microservice Vulnerabilities (20%)

Kubernetes Secrets are the native mechanism for storing sensitive data — passwords, tokens, keys, certificates. But “native” does not mean “secure by default.” Secrets are base64-encoded, not encrypted, and stored in etcd as plaintext. Understanding exactly how Secrets work — and where they fail — is essential for the CKS exam.

1. How Secrets Are Stored in etcd

When you create a Secret, the API server writes it to etcd. By default, the data is stored base64-encoded — which is an encoding, not encryption. Anyone with direct etcd access can decode every Secret in the cluster.

kubectl create secret → API Server → etcd
                                      │
                                      ▼
                        /registry/secrets/<namespace>/<name>
                        │
                        ├── key: "password"
                        └── value: "cGFzc3dvcmQxMjM="  ← base64 of "password123"

Base64 is trivially reversible:

echo "cGFzc3dvcmQxMjM=" | base64 -d
# password123

Security Impact: Without encryption at rest, an attacker who gains access to etcd backups, etcd data directories, or etcd API endpoints has access to every Secret in your cluster.

2. Secret Types

Kubernetes defines several built-in Secret types. Each type enforces specific data key requirements.

Type Description Required Keys Use Case
Opaque Default type, arbitrary key-value pairs None (any keys) Generic passwords, API keys
kubernetes.io/dockerconfigjson Docker registry credentials .dockerconfigjson Image pull authentication
kubernetes.io/tls TLS certificate and key pair tls.crt, tls.key Ingress TLS, mutual TLS
kubernetes.io/basic-auth Basic authentication credentials username, password HTTP basic auth
kubernetes.io/ssh-auth SSH private key ssh-privatekey Git operations, SSH tunnels
kubernetes.io/service-account-token ServiceAccount token (legacy) Auto-generated API authentication

Creating Different Types

# Opaque (default)
kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=s3cur3P@ss

# Docker registry
kubectl create secret docker-registry reg-cred \
  --docker-server=registry.example.com \
  --docker-username=admin \
  --docker-password=dckr_pat_xxxx \
  --docker-email=admin@example.com

# TLS
kubectl create secret tls app-tls \
  --cert=tls.crt \
  --key=tls.key

# From file
kubectl create secret generic ssh-key \
  --from-file=ssh-privatekey=/path/to/id_rsa

3. Consuming Secrets — Environment Variables vs Volume Mounts

There are two primary ways Pods consume Secrets. They are not equivalent from a security perspective.

Option A: Environment Variables

apiVersion: v1
kind: Pod
metadata:
  name: env-secret-pod
spec:
  containers:
  - name: app
    image: nginx:1.25
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-creds
          key: password

Option B: Volume Mounts

apiVersion: v1
kind: Pod
metadata:
  name: vol-secret-pod
spec:
  containers:
  - name: app
    image: nginx:1.25
    volumeMounts:
    - name: secret-vol
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secret-vol
    secret:
      secretName: db-creds

Security Comparison

Factor Environment Variables Volume Mounts
Visibility in kubectl describe Yes — shown in Pod spec No — only mount path shown
Exposure in crash dumps Yes — env is part of process info No — files on disk
Exposure in application logs High risk — logging frameworks dump env Lower risk — must explicitly read file
Available to child processes Yes — env inherited by all children No — must explicitly read files
Update without Pod restart No Yes — kubelet refreshes mounted Secrets
Usable by envFrom (bulk inject) Yes — but unsafe, injects ALL keys N/A

CKS Best Practice: Always prefer volume mounts over environment variables for Secret consumption. Environment variables leak in too many ways.

4. Immutable Secrets

Kubernetes supports marking Secrets as immutable (stable since v1.21). Once set, the Secret cannot be updated — only deleted and recreated.

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
  namespace: production
type: Opaque
data:
  username: YWRtaW4=       # admin
  password: czNjdXIzUEBzcw== # s3cur3P@ss
immutable: true

Benefits

  • Prevents accidental or malicious modification of critical credentials
  • Reduces etcd watch load — the API server stops tracking changes for immutable Secrets
  • Improves cluster performance at scale — fewer watches = less etcd CPU

Limitations

  • To rotate an immutable Secret, you must delete it and create a new one
  • Pods referencing the Secret must be restarted to pick up the new version
  • Cannot be “un-immutabled” — the field is one-way
# Trying to update an immutable secret fails:
kubectl edit secret db-creds
# Error: secret "db-creds" is immutable

# To rotate: delete and recreate
kubectl delete secret db-creds
kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=n3wP@ssw0rd

5. Secret Risks — Attack Vectors

Risk 1: etcd Direct Access

# If an attacker has etcd credentials, they can read all secrets:
ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-creds \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Output contains base64-encoded secret data in plaintext

Mitigation: Enable encryption at rest (next topic in this chapter).

Risk 2: Overpermissioned RBAC

# Check who can list all secrets in a namespace:
kubectl auth can-i list secrets --as=system:serviceaccount:default:default -n default
# yes ← DANGEROUS — this SA can read every secret in the namespace

# Check who can get secrets cluster-wide:
kubectl auth can-i get secrets --all-namespaces --as=developer

Mitigation: Restrict RBAC — never grant list or watch on secrets unless absolutely required.

Risk 3: Environment Variable Exposure

# Anyone who can describe a pod sees env-injected secrets:
kubectl describe pod env-secret-pod
# ...
# Environment:
#   DB_PASSWORD:  <set to the key 'password' in secret 'db-creds'>

While kubectl describe doesn’t show the value directly, kubectl get pod -o yaml does:

kubectl get pod env-secret-pod -o jsonpath='{.spec.containers[0].env}'
# Shows the secretKeyRef — and anyone with get-pods + get-secrets can read the actual value

Mitigation: Use volume mounts. Restrict Pod get permissions to trusted identities.

Risk 4: Logging and Debug Endpoints

Applications that log all environment variables will dump secrets:

# Dangerous Python code:
import os
print(os.environ)  # Logs DB_PASSWORD to stdout

# Dangerous shell pattern:
env | tee /tmp/debug.log  # Writes all env vars including secrets

Mitigation: Never log environment variables. Use volume-mounted secrets and read files directly.

Risk 5: etcd Backups

etcd snapshots contain all Secrets in plaintext (unless encryption at rest is enabled). A stolen backup = a stolen credential store.

# etcd snapshot contains everything:
ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-backup.db
# /tmp/etcd-backup.db contains ALL secrets in base64

Mitigation: Encrypt etcd backups. Enable encryption at rest. Restrict access to backup storage.

6. Creating Secrets — Exam Commands

These are the commands you need to know by heart for the CKS exam:

# Create from literals
kubectl create secret generic my-secret \
  --from-literal=key1=value1 \
  --from-literal=key2=value2

# Create from file (key name = filename by default)
kubectl create secret generic my-secret \
  --from-file=config.json

# Create from file with custom key name
kubectl create secret generic my-secret \
  --from-file=app-config=config.json

# Create in a specific namespace
kubectl create secret generic my-secret \
  --from-literal=password=secret123 \
  -n production

# Create from env file
kubectl create secret generic my-secret \
  --from-env-file=secrets.env

# Dry run + YAML output (for manifests)
kubectl create secret generic my-secret \
  --from-literal=password=secret123 \
  --dry-run=client -o yaml > secret.yaml

7. Viewing and Decoding Secrets

# List secrets in a namespace
kubectl get secrets -n default

# View secret metadata (does NOT show data)
kubectl describe secret db-creds

# View base64-encoded data
kubectl get secret db-creds -o yaml

# Decode a specific key using jsonpath
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
# s3cur3P@ss

# Decode all keys
kubectl get secret db-creds -o json | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'
# username: admin
# password: s3cur3P@ss

# Quick one-liner for a single field
kubectl get secret db-creds -o go-template='{{ .data.password | base64decode }}'

8. ServiceAccount Token Secrets

Prior to Kubernetes v1.24, creating a ServiceAccount automatically generated a long-lived Secret containing a JWT token. This was a security risk — the token never expired.

Legacy Behavior (pre-1.24)

# Auto-created secret:
apiVersion: v1
kind: Secret
metadata:
  name: my-sa-token-xxxxx
  annotations:
    kubernetes.io/service-account.name: my-sa
type: kubernetes.io/service-account-token
data:
  token: <base64-encoded JWT>
  ca.crt: <base64-encoded CA>
  namespace: <base64-encoded namespace>

Current Behavior (v1.24+)

  • Tokens are no longer auto-created as Secrets
  • The TokenRequest API issues short-lived, audience-bound tokens
  • Tokens are projected into Pods via the projected volume (default)
  • Long-lived tokens can still be created manually if needed (but avoid this)
# Create a time-limited token (preferred):
kubectl create token my-sa --duration=1h

# Create a legacy long-lived token (avoid in production):
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: my-sa-token
  annotations:
    kubernetes.io/service-account.name: my-sa
type: kubernetes.io/service-account-token
EOF

CKS Exam Tip: Know that projected service account tokens (short-lived, audience-bound) are the secure default since v1.24. Legacy Secrets-based tokens are a security risk.

Summary

Aspect Key Takeaway
Storage Secrets are base64-encoded in etcd — not encrypted by default
Consumption Prefer volume mounts over environment variables
Immutability Use immutable: true for critical secrets
RBAC Never grant list/watch on secrets to unnecessary identities
SA Tokens Use TokenRequest API (v1.24+) — avoid long-lived token Secrets
Encryption Enable encryption at rest (next topic)

Practice Questions

Question 1

What encoding format does Kubernetes use to store Secret values in etcd by default?

Show Answer

Base64 encoding. Kubernetes does NOT encrypt Secrets by default. The values in .data fields are base64-encoded, which is trivially reversible:

echo "cGFzc3dvcmQxMjM=" | base64 -d
# password123

This means anyone with etcd access can read all Secrets. To actually encrypt Secrets, you must configure encryption at rest via EncryptionConfiguration.

Question 2

Why are volume-mounted Secrets preferred over environment-variable Secrets from a security perspective?

Show Answer

Volume-mounted Secrets are preferred because:

  1. Environment variables leak — they appear in kubectl describe pod, crash dumps, child process environments, and logging framework output.
  2. Volume mounts are opaque — only the mount path is visible in the Pod spec, not the values.
  3. Volume mounts auto-update — when a Secret changes, the kubelet refreshes the mounted files (with a delay). Environment variables require a Pod restart.
  4. Environment variables propagate to child processes — any subprocess inherits the parent environment, including secrets.
# Preferred:
volumeMounts:
- name: secret-vol
  mountPath: /etc/secrets
  readOnly: true

# Avoid:
env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-creds
      key: password

Question 3

What does immutable: true do on a Secret, and what are its benefits?

Show Answer

Setting immutable: true on a Secret means:

  • The Secret cannot be updated after creation (only deleted and recreated)
  • The API server stops watching the Secret for changes, reducing etcd load
  • This protects against accidental or malicious modification of credentials
apiVersion: v1
kind: Secret
metadata:
  name: critical-creds
type: Opaque
data:
  api-key: c3VwZXJzZWNyZXQ=
immutable: true

To rotate an immutable Secret, you must delete it and create a new one, then restart Pods that reference it.

Question 4

A team grants their CI/CD ServiceAccount list and watch permissions on Secrets in the production namespace. Why is this dangerous?

Show Answer

list and watch on Secrets means the ServiceAccount can read every Secret in the namespace, not just specific ones. There is no way to scope list/watch to individual Secret names in RBAC.

# DANGEROUS — grants access to ALL secrets in production:
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["list", "watch"]

If the CI/CD pipeline or its Pod is compromised, the attacker gets every credential in production.

Better approach: Grant get on specific Secret resourceNames:

rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["ci-deploy-token"]
  verbs: ["get"]

Question 5

How do projected ServiceAccount tokens (v1.24+) improve security over legacy Secret-based SA tokens?

Show Answer

Projected tokens (via the TokenRequest API) are more secure because:

  1. Time-limited — they expire (default: 1 hour, configurable). Legacy tokens never expire.
  2. Audience-bound — they specify which API server they’re valid for. Legacy tokens work against any API server.
  3. Not stored as Secrets — they’re generated on-demand and projected into the Pod. No Secret object exists in etcd.
  4. Auto-rotated — the kubelet refreshes the token before it expires.
# Short-lived token (preferred):
kubectl create token my-sa --duration=1h

# Legacy token (avoid):
# Creates a persistent Secret in etcd that never expires

For the CKS exam, know that v1.24+ uses the TokenRequest API by default, and creating long-lived SA token Secrets is a security anti-pattern.

Previous Up Next
Secrets Management Secrets Management Encryption at Rest

Updated:

You may also enjoy