CKS — Secret Rotation

10 minute read

Secret Rotation

CKS Exam Domain: Minimize Microservice Vulnerabilities (20%)

Secrets have a shelf life. The longer a credential exists unchanged, the wider the window for compromise — through leaked logs, stolen backups, social engineering, or brute force. Secret rotation reduces this window by periodically replacing credentials with new ones.

1. Why Rotate Secrets?

The Compromise Window

Secret Created ──────────────────────────────────────── Secret Compromised
       │                                                        │
       │◄──────── Compromise Window (exposure time) ──────────►│
       │                                                        │
       │  Longer window = more damage                           │
       │  Shorter window = less damage                          │
       │                                                        │
       ▼                                                        ▼
  If you rotate every 30 days:
  Maximum exposure = 30 days (even if you don't know about the breach)

  If you NEVER rotate:
  Maximum exposure = forever

Rotation Benefits

Benefit Description
Reduced blast radius Compromised credentials expire before they can be fully exploited
Compliance requirements PCI DSS, SOC 2, HIPAA mandate periodic rotation
Limits lateral movement Attackers with old credentials lose access
Forces good hygiene Rotation workflows ensure secrets are properly managed
Credential revocation Old credentials stop working after rotation

2. Manual Rotation Workflow

For native Kubernetes Secrets without external tooling, rotation is a manual process.

Step 1: Generate New Credentials

# Example: generate a new database password
NEW_PASSWORD=$(openssl rand -base64 24)
echo "New password: $NEW_PASSWORD"

Step 2: Update the External System

Update the database, API, or service to accept the new credential:

# Example: update PostgreSQL password
psql -U postgres -c "ALTER USER appuser PASSWORD '${NEW_PASSWORD}';"

Step 3: Update the Kubernetes Secret

# Option A: Delete and recreate
kubectl delete secret db-creds -n production
kubectl create secret generic db-creds \
  --from-literal=password="${NEW_PASSWORD}" \
  -n production

# Option B: Patch in place (keeps metadata, labels, annotations)
kubectl create secret generic db-creds \
  --from-literal=password="${NEW_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

Step 4: Restart Pods to Pick Up New Secret

# Rolling restart of the deployment
kubectl rollout restart deployment/myapp -n production

# Verify rollout
kubectl rollout status deployment/myapp -n production

Step 5: Verify

# Confirm new Pods are running
kubectl get pods -n production -l app=myapp

# Confirm the secret has the new value
kubectl get secret db-creds -n production -o jsonpath='{.data.password}' | base64 -d

Manual Rotation Risks

  • Human error — someone forgets to update the external system first
  • Downtime — brief period where Pods have the old credential and the database expects the new one
  • Forgotten secrets — secrets that nobody remembers to rotate
  • No audit trail — hard to prove when secrets were last rotated

3. Automated Rotation with External Stores

HashiCorp Vault — Dynamic Secrets

Vault’s dynamic secrets are the gold standard for automated rotation. Vault generates credentials on-demand with a lease — when the lease expires, Vault automatically revokes the credentials.

Pod Starts → Vault Agent → Vault → Database
                │                    │
                │ Request creds      │ Create user/pass
                │◄──────────────────│ (TTL: 1h)
                │                    │
                │ Write to          │
                │ /vault/secrets/   │
                │                    │
    ... 1 hour later ...             │
                │                    │
                │ Renew lease       │ Extend TTL
                │◄──────────────────│
                │                    │
    ... or if lease expires ...      │
                │                    │
                │                    │ Revoke creds
                │ Get new creds     │ Create new user/pass
                │◄──────────────────│

Vault Dynamic Database Credentials

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/mydb \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role" \
  connection_url="postgresql://:@db.example.com:5432/mydb" \
  username="vault_admin" \
  password="vault_password"

# Create a role with a TTL
vault write database/roles/app-role \
  db_name=mydb \
  creation_statements="CREATE ROLE \"\" WITH LOGIN PASSWORD '' VALID UNTIL ''; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Generate credentials (done automatically by Vault Agent)
vault read database/creds/app-role
# username: v-kubernetes-app-role-xxxxx
# password: A1a-xxxxxxxxxxxxxxxxxx
# lease_duration: 1h

AWS Secrets Manager — Rotation Lambda

AWS Secrets Manager can automatically rotate secrets using a Lambda function:

{
  "rotation_rules": {
    "automatically_after_days": 30
  }
}

The rotation Lambda:

  1. Creates a new credential version (AWSPENDING)
  2. Updates the external service (e.g., RDS password)
  3. Tests the new credential
  4. Promotes it to AWSCURRENT

4. Restarting Pods After Secret Update

When a Kubernetes Secret is updated, Pods don’t automatically restart. Depending on how the secret is consumed, the behavior differs:

Volume-Mounted Secrets — Auto-Refresh

If secrets are mounted as volumes, the kubelet periodically refreshes the mounted files (default sync period: ~1 minute, configurable via --sync-frequency).

# Pods with volume mounts get updates automatically (eventually):
volumes:
- name: secret-vol
  secret:
    secretName: db-creds

Caveat: The application must re-read the file. If it reads the secret once at startup and caches it in memory, the refresh is useless.

Environment Variable Secrets — Require Pod Restart

Environment variables are set at container start and never change during the container’s lifetime:

# Pods with env vars MUST be restarted to get new values:
env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-creds
      key: password

Rolling Restart

# Restart all Pods in a Deployment
kubectl rollout restart deployment/myapp -n production

# Restart all Pods in a StatefulSet
kubectl rollout restart statefulset/mydb -n production

# Restart all Pods in a DaemonSet
kubectl rollout restart daemonset/agent -n production

Stakater Reloader — Automatic Restart on Secret Change

Stakater Reloader is an open-source controller that watches for Secret/ConfigMap changes and automatically triggers rolling restarts.

# Annotate the Deployment to trigger restart on secret change:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  annotations:
    reloader.stakater.com/auto: "true"   # Watch ALL referenced Secrets/ConfigMaps
    # OR be specific:
    # secret.reloader.stakater.com/reload: "db-creds"
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:1.0
        envFrom:
        - secretRef:
            name: db-creds

When db-creds is updated, Reloader automatically triggers a rolling restart of the Deployment.

How Reloader Works

  1. Watches for changes to Secrets and ConfigMaps
  2. Computes a hash of the Secret/ConfigMap data
  3. When the hash changes, patches the Deployment/StatefulSet/DaemonSet with a new annotation
  4. The annotation change triggers a rolling update

5. Projected Volumes for Automatic Refresh

Projected volumes combine multiple volume sources into a single mount point and support automatic token refresh for ServiceAccount tokens.

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: combined
      mountPath: /etc/config
      readOnly: true
  volumes:
  - name: combined
    projected:
      sources:
      - secret:
          name: db-creds
          items:
          - key: password
            path: db-password
      - configMap:
          name: app-config
          items:
          - key: config.yaml
            path: config.yaml
      - serviceAccountToken:
          path: token
          expirationSeconds: 3600    # Auto-rotated by kubelet
          audience: "https://myapi.example.com"

Benefits of Projected Volumes

  • ServiceAccount tokens are auto-rotated — kubelet refreshes before expiration
  • Audience-bound — tokens specify who they’re intended for
  • Combined mount — secrets, configmaps, and tokens in one directory
  • Expiration-aware — configurable TTL for SA tokens

6. Immutable Secrets and Rotation

Immutable Secrets (immutable: true) cannot be updated in place. This changes the rotation workflow:

Rotation with Immutable Secrets

# Step 1: Create the new secret with a different name
kubectl create secret generic db-creds-v2 \
  --from-literal=password=newP@ssw0rd \
  -n production

# Make it immutable
kubectl patch secret db-creds-v2 -n production \
  --type merge -p '{"immutable": true}'

# Step 2: Update the Deployment to reference the new secret
kubectl set env deployment/myapp \
  --from=secret/db-creds-v2 -n production
# OR edit the Deployment to change the secret name in volumes

# Step 3: Verify Pods restart with new secret
kubectl rollout status deployment/myapp -n production

# Step 4: Delete the old secret
kubectl delete secret db-creds -n production

Alternative: Blue-Green Secret Rotation

# Use two secrets with a consistent naming convention:
# db-creds-blue  (current)
# db-creds-green (next rotation)

# Deployment references a label or specific secret name:
volumes:
- name: db-creds
  secret:
    secretName: db-creds-blue    # Switch to db-creds-green during rotation

7. Rotation Strategy Comparison

Strategy Automation Downtime Risk Complexity Best For
Manual delete + recreate None High (gap between delete and create) Low Dev/test
Manual kubectl apply None Low (atomic update) Low Small production
Stakater Reloader Auto restart on change Low Medium Production without Vault
Vault dynamic secrets Full auto (lease-based) None High Enterprise production
AWS SM rotation Lambda Full auto (scheduled) Low Medium AWS-native production
Immutable + rename Manual Low (no gap) Medium Compliance-heavy environments

8. Monitoring Secret Rotation

Check Secret Age

# List secrets with creation timestamp
kubectl get secrets -n production -o custom-columns=\
'NAME:.metadata.name,CREATED:.metadata.creationTimestamp'

# Find secrets older than 90 days
kubectl get secrets -A -o json | jq -r '
  .items[] |
  select(.metadata.creationTimestamp |
    fromdateiso8601 < (now - 7776000)) |
  "\(.metadata.namespace)/\(.metadata.name) created \(.metadata.creationTimestamp)"
'

Audit Secret Access

# Who accessed secrets recently (from audit logs):
grep '"secrets"' /var/log/kubernetes/audit.log | \
  jq -r '[.user.username, .verb, .objectRef.namespace + "/" + .objectRef.name] | @tsv' | \
  sort | uniq -c | sort -rn

Summary

Aspect Key Takeaway
Rotation necessity Reduces compromise window; required by compliance
Manual rotation Delete + recreate, or kubectl apply; restart Pods
Vault dynamic secrets Best automation — lease-based, auto-revoke
Volume-mounted secrets Auto-refresh by kubelet (~1 min); app must re-read
Env var secrets No auto-refresh — Pod restart required
Stakater Reloader Auto-restarts on Secret change (annotation-based)
Immutable secrets Must create new Secret + update Pod spec

Practice Questions

Question 1

A Deployment uses environment variables to inject a Secret. You update the Secret value. Why don’t the Pods see the new value?

Show Answer

Environment variables are set at container start and never change during the container’s lifetime. Updating the underlying Secret does NOT update the environment variable in running containers.

To apply the new Secret value, you must restart the Pods:

kubectl rollout restart deployment/myapp -n production

This triggers a rolling update, creating new Pods that read the updated Secret at startup.

Better approach: Use volume mounts instead of env vars. Volume-mounted Secrets are periodically refreshed by the kubelet (default ~1 minute sync) without requiring a Pod restart. However, the application must re-read the file to pick up changes.

Question 2

What is the advantage of Vault’s dynamic secrets over manual secret rotation?

Show Answer

Vault dynamic secrets are superior because:

  1. On-demand generation — credentials are created when requested, not pre-provisioned
  2. Lease-based — each credential has a TTL (e.g., 1 hour); when it expires, Vault automatically revokes it
  3. Unique per consumer — each Pod gets its own credential, making forensics easier
  4. No shared credentials — if one Pod is compromised, only its credential is exposed
  5. Automatic revocation — no manual cleanup; Vault revokes expired credentials automatically
# Vault generates unique creds per request:
vault read database/creds/app-role
# username: v-kubernetes-app-role-abc123
# password: A1a-xxxxxxxxxx
# lease_duration: 1h
# lease_id: database/creds/app-role/xxxxx

With manual rotation, you have one shared credential that all Pods use, and rotation requires manual effort plus Pod restarts.

Question 3

How does Stakater Reloader trigger a Pod restart when a Secret changes?

Show Answer

Stakater Reloader works by:

  1. Watching Kubernetes Secrets and ConfigMaps for changes
  2. Computing a hash of the Secret/ConfigMap data
  3. When the hash changes, Reloader patches the Deployment (or StatefulSet/DaemonSet) by adding or updating an annotation with the new hash
  4. The annotation change triggers the Deployment controller to perform a rolling update
# Before secret change:
metadata:
  annotations:
    reloader.stakater.com/auto: "true"

# After secret change, Reloader adds:
spec:
  template:
    metadata:
      annotations:
        secret.reloader.stakater.com/reload-hash: "abc123def456"

This is transparent to the application — it just sees a normal rolling restart. The Deployment retains its rollout history and can be rolled back if needed.

Question 4

You have an immutable Secret (immutable: true) that needs to be rotated. What is the correct procedure?

Show Answer

Immutable Secrets cannot be updated in place. The rotation procedure is:

  1. Create a new Secret with a different name: 
    kubectl create secret generic db-creds-v2 \
  --from-literal=password=new-password -n production
kubectl patch secret db-creds-v2 -n production \
  --type merge -p '{"immutable": true}'
  2. Update the workload to reference the new Secret: 
    # Edit Deployment to change secretName from db-creds to db-creds-v2
kubectl edit deployment/myapp -n production
  3. Verify the rollout: 
    kubectl rollout status deployment/myapp -n production
  4. Delete the old Secret: 
    kubectl delete secret db-creds -n production

This is a deliberate trade-off: immutable Secrets are safer (no accidental overwrites), but rotation requires extra steps.

Question 5

A volume-mounted Secret is updated, but the application doesn’t see the new value. The kubelet sync has already occurred. What’s wrong?

Show Answer

The application is caching the secret value in memory after reading it once at startup. Even though the kubelet has refreshed the mounted file on disk, the application isn’t re-reading it.

The kubelet sync only updates the file on the filesystem. It’s the application’s responsibility to:

  1. Watch the file for changes (using inotify, fsnotify, etc.)
  2. Periodically re-read the file
  3. Use a signal handler to trigger a config reload (e.g., SIGHUP)
# Verify the file has been updated on disk:
kubectl exec -it myapp-pod -- cat /etc/secrets/password
# Shows the NEW value — the kubelet did its job

# The application is using the OLD in-memory value
# Fix: design the app to re-read the file periodically

Note: The kubelet sync interval is configurable via --sync-frequency (default: 1 minute). There may also be an additional TTL-based cache delay.

Previous Up Next
External Secret Stores Secrets Management Secret Access Control

Updated:

You may also enjoy