CKS — Secrets Management Lab

13 minute read

Secrets Management — Hands-On Lab

CKS Exam Domain: Minimize Microservice Vulnerabilities (20%)

This lab walks you through the complete lifecycle of secrets management in Kubernetes — from creating secrets to encrypting them at rest, restricting access with RBAC, and auditing permissions. Each exercise mirrors the type of tasks you may face on the CKS exam.

Prerequisites

  • A Kubernetes cluster with control plane access (kubeadm preferred — you need access to API server manifests and etcd)
  • kubectl configured with cluster-admin access
  • etcdctl installed on the control plane node
  • SSH access to the control plane node (for modifying static Pod manifests)

Lab 1: Create Secrets Using kubectl

Task

Create Kubernetes Secrets using different methods — from literals, from files, and verify their contents.

Steps

# Create a namespace for the lab
kubectl create namespace secrets-lab

# 1. Create a secret from literal values
kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=CKS-Lab-P@ss123 \
  -n secrets-lab

# 2. Create a secret from a file
echo -n "my-api-key-value-12345" > /tmp/api-key.txt
kubectl create secret generic api-key \
  --from-file=key=/tmp/api-key.txt \
  -n secrets-lab

# 3. Create a TLS secret (generate self-signed cert first)
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout /tmp/tls.key -out /tmp/tls.crt \
  -subj "/CN=lab.example.com"

kubectl create secret tls lab-tls \
  --cert=/tmp/tls.crt \
  --key=/tmp/tls.key \
  -n secrets-lab

# 4. Verify secrets were created
kubectl get secrets -n secrets-lab

Expected Output

NAME        TYPE                       DATA   AGE
api-key     Opaque                     1      5s
db-creds    Opaque                     2      10s
lab-tls     kubernetes.io/tls          2      3s

Verify Secret Contents

# Decode the db-creds secret
kubectl get secret db-creds -n secrets-lab -o jsonpath='{.data.username}' | base64 -d
# admin

kubectl get secret db-creds -n secrets-lab -o jsonpath='{.data.password}' | base64 -d
# CKS-Lab-P@ss123

# Decode the api-key secret
kubectl get secret api-key -n secrets-lab -o jsonpath='{.data.key}' | base64 -d
# my-api-key-value-12345

# View all keys in a secret
kubectl get secret db-creds -n secrets-lab -o json | jq -r '.data | keys[]'
# password
# username

Lab 2: Verify Secrets Are Base64 in etcd (Before Encryption)

Task

Prove that Secrets are stored as base64-encoded plaintext in etcd before encryption at rest is enabled.

Steps

# SSH to the control plane node (if not already there)

# Read the db-creds secret directly from etcd
ETCDCTL_API=3 etcdctl get /registry/secrets/secrets-lab/db-creds \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Expected Output

You should see the Secret data in readable form within the output. Look for strings like admin and CKS-Lab-P@ss123 in the raw output.

# Use hexdump for clearer output
ETCDCTL_API=3 etcdctl get /registry/secrets/secrets-lab/db-creds \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C | head -40

# You will see ASCII representations of the secret values
# This proves secrets are NOT encrypted in etcd by default

What This Proves

  • Secrets in etcd are base64-encoded, not encrypted
  • Anyone with etcd credentials can read every secret in the cluster
  • etcd backups contain all secrets in plaintext
  • Encryption at rest is essential for production security

Lab 3: Create EncryptionConfiguration for aescbc

Task

Create an EncryptionConfiguration resource that encrypts Secrets in etcd using the aescbc provider.

Steps

# SSH to the control plane node

# Step 1: Generate a 32-byte encryption key
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
echo "Generated key: ${ENCRYPTION_KEY}"

# Step 2: Create the directory
sudo mkdir -p /etc/kubernetes/enc

# Step 3: Create the EncryptionConfiguration
cat <<EOF | sudo tee /etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

# Step 4: Set restrictive permissions
sudo chmod 600 /etc/kubernetes/enc/enc.yaml
sudo chown root:root /etc/kubernetes/enc/enc.yaml

# Step 5: Verify the file
sudo cat /etc/kubernetes/enc/enc.yaml

Expected Output

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <your-generated-key>
      - identity: {}

Verification Checklist

  • File exists at /etc/kubernetes/enc/enc.yaml
  • Permissions are 600 (readable only by root)
  • aescbc is the first provider (before identity)
  • identity: {} is present as fallback for reading old unencrypted secrets
  • Key is exactly 32 bytes base64-encoded

Lab 4: Configure API Server with –encryption-provider-config

Task

Modify the kube-apiserver static Pod manifest to use the encryption configuration.

Steps

# Step 1: Back up the current API server manifest
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml \
  /etc/kubernetes/manifests/kube-apiserver.yaml.bak

# Step 2: Edit the manifest
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Changes to Make

Add the flag to the command section:

spec:
  containers:
  - command:
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml   # ADD THIS LINE
    # ... keep all existing flags ...

Add the volume mount to the container:

    volumeMounts:
    - name: enc-config                   # ADD THIS BLOCK
      mountPath: /etc/kubernetes/enc
      readOnly: true
    # ... keep all existing volume mounts ...

Add the volume to the Pod spec:

  volumes:
  - name: enc-config                     # ADD THIS BLOCK
    hostPath:
      path: /etc/kubernetes/enc
      type: DirectoryOrCreate
  # ... keep all existing volumes ...

Wait for Restart

# The kubelet will detect the change and restart the API server
# Wait for it to come back (may take 30-60 seconds):

# Method 1: Watch kubectl
watch -n 2 "kubectl get nodes 2>&1"

# Method 2: Watch the container
sudo crictl ps | grep kube-apiserver

# Method 3: Check logs if it doesn't come back
sudo crictl logs $(sudo crictl ps -a | grep kube-apiserver | head -1 | awk '{print $1}') 2>&1 | tail -20

Troubleshooting

If the API server doesn’t come back:

# Check for common errors:

# 1. File not found — verify the path
sudo ls -la /etc/kubernetes/enc/enc.yaml

# 2. YAML syntax error — validate
python3 -c "import yaml; yaml.safe_load(open('/etc/kubernetes/enc/enc.yaml'))"

# 3. Key length error — verify key is 32 bytes
KEY=$(sudo grep 'secret:' /etc/kubernetes/enc/enc.yaml | awk '{print $2}')
echo -n "$KEY" | base64 -d | wc -c
# Must be exactly 32

# 4. Restore backup if needed
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml.bak \
  /etc/kubernetes/manifests/kube-apiserver.yaml

Lab 5: Verify Encryption in etcd (After Encryption)

Task

Create a new Secret after encryption is enabled and verify it’s encrypted in etcd.

Steps

# Step 1: Create a new test secret
kubectl create secret generic encrypted-test \
  --from-literal=testkey=testvalue123 \
  -n secrets-lab

# Step 2: Read it from etcd
ETCDCTL_API=3 etcdctl get /registry/secrets/secrets-lab/encrypted-test \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

Expected Output

You should see the prefix k8s:enc:aescbc:v1:key1 followed by binary (encrypted) data. The actual secret value testvalue123 should NOT be visible.

00000000  ... k 8 s : e n c : a e s c b c : v 1 : k e y 1 ...
00000010  ... <binary encrypted data> ...

Compare: Old Secret (Before Encryption)

# The db-creds secret was created BEFORE encryption was enabled
# It should still be in plaintext:
ETCDCTL_API=3 etcdctl get /registry/secrets/secrets-lab/db-creds \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

# You should still see "admin" and "CKS-Lab-P@ss123" in plaintext
# This proves that enabling encryption does NOT retroactively encrypt existing secrets

Verify via kubectl (Should Still Work)

# kubectl reads through the API server, which handles decryption transparently:
kubectl get secret encrypted-test -n secrets-lab -o jsonpath='{.data.testkey}' | base64 -d
# testvalue123

Lab 6: Encrypt All Existing Secrets

Task

Re-encrypt all pre-existing Secrets so they are stored encrypted in etcd.

Steps

# Step 1: Re-encrypt all secrets in all namespaces
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Expected Output

You’ll see each Secret being replaced:

secret/db-creds replaced
secret/api-key replaced
secret/lab-tls replaced
secret/encrypted-test replaced
...

Verify the Old Secret Is Now Encrypted

# Check db-creds which was created BEFORE encryption:
ETCDCTL_API=3 etcdctl get /registry/secrets/secrets-lab/db-creds \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

# NOW you should see k8s:enc:aescbc:v1:key1 prefix
# The plaintext values should NO LONGER be visible

Verify All Secrets Still Work

# Confirm secrets are still readable through the API:
kubectl get secret db-creds -n secrets-lab -o jsonpath='{.data.password}' | base64 -d
# CKS-Lab-P@ss123

kubectl get secret api-key -n secrets-lab -o jsonpath='{.data.key}' | base64 -d
# my-api-key-value-12345

Lab 7: Create RBAC to Restrict Secret Access

Task

Create RBAC rules that allow a ServiceAccount to read only a specific Secret.

Steps

# Step 1: Create a ServiceAccount
kubectl create serviceaccount app-sa -n secrets-lab

# Step 2: Create a Role that only allows reading db-creds
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: db-creds-reader
  namespace: secrets-lab
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["db-creds"]
  verbs: ["get"]
EOF

# Step 3: Create a RoleBinding
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-sa-read-db-creds
  namespace: secrets-lab
subjects:
- kind: ServiceAccount
  name: app-sa
  namespace: secrets-lab
roleRef:
  kind: Role
  name: db-creds-reader
  apiGroup: rbac.authorization.k8s.io
EOF

Verify Permissions

# Can get db-creds? (should be YES)
kubectl auth can-i get secrets/db-creds \
  --as=system:serviceaccount:secrets-lab:app-sa \
  -n secrets-lab
# yes

# Can get api-key? (should be NO)
kubectl auth can-i get secrets/api-key \
  --as=system:serviceaccount:secrets-lab:app-sa \
  -n secrets-lab
# no

# Can list all secrets? (should be NO)
kubectl auth can-i list secrets \
  --as=system:serviceaccount:secrets-lab:app-sa \
  -n secrets-lab
# no

# Can create secrets? (should be NO)
kubectl auth can-i create secrets \
  --as=system:serviceaccount:secrets-lab:app-sa \
  -n secrets-lab
# no

Test Access from a Pod

# Create a Pod using the app-sa ServiceAccount
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: rbac-test
  namespace: secrets-lab
spec:
  serviceAccountName: app-sa
  automountServiceAccountToken: true
  containers:
  - name: test
    image: bitnami/kubectl:latest
    command: ["sleep", "3600"]
EOF

# Wait for Pod to be running
kubectl wait --for=condition=ready pod/rbac-test -n secrets-lab --timeout=60s

# Test from inside the Pod:

# Should SUCCEED:
kubectl exec -it rbac-test -n secrets-lab -- \
  kubectl get secret db-creds -n secrets-lab -o jsonpath='{.data.password}'
# Q0tTLUxhYi1QQHNzMTIz  (base64)

# Should FAIL (Forbidden):
kubectl exec -it rbac-test -n secrets-lab -- \
  kubectl get secret api-key -n secrets-lab 2>&1
# Error from server (Forbidden): secrets "api-key" is forbidden

# Should FAIL (Forbidden):
kubectl exec -it rbac-test -n secrets-lab -- \
  kubectl get secrets -n secrets-lab 2>&1
# Error from server (Forbidden): secrets is forbidden

Lab 8: Audit Who Can Access Secrets

Task

Perform a comprehensive audit of secret access permissions in the cluster.

Steps

# Step 1: Audit ServiceAccounts in secrets-lab namespace
echo "=== Secret Access Audit: secrets-lab ==="
for sa in $(kubectl get sa -n secrets-lab -o jsonpath='{.items[*].metadata.name}'); do
  echo ""
  echo "ServiceAccount: secrets-lab/$sa"
  for verb in get list watch create update delete; do
    result=$(kubectl auth can-i $verb secrets \
      --as=system:serviceaccount:secrets-lab:$sa \
      -n secrets-lab 2>/dev/null)
    if [ "$result" = "yes" ]; then
      echo "  ✗ CAN $verb secrets"
    fi
  done
done

Expected Output

=== Secret Access Audit: secrets-lab ===

ServiceAccount: secrets-lab/default
  (no secret permissions — good!)

ServiceAccount: secrets-lab/app-sa
  ✗ CAN get secrets    ← Expected (scoped to db-creds only)

Step 2: Cluster-Wide Secret Access Audit

# Find all ClusterRoleBindings that grant secret access:
echo "=== Cluster-Wide Secret Access ==="
for crb in $(kubectl get clusterrolebindings -o jsonpath='{.items[*].metadata.name}'); do
  role=$(kubectl get clusterrolebinding "$crb" -o jsonpath='{.roleRef.name}')

  # Check if the role has secret access
  has_secrets=$(kubectl get clusterrole "$role" -o json 2>/dev/null | \
    jq -r '.rules[]? | select(.resources[]? == "secrets" or .resources[]? == "*") | .verbs[]' 2>/dev/null)

  if [ -n "$has_secrets" ]; then
    subjects=$(kubectl get clusterrolebinding "$crb" \
      -o jsonpath='{range .subjects[*]}{.kind}/{.name} {end}')
    echo ""
    echo "ClusterRoleBinding: $crb"
    echo "  Role: $role"
    echo "  Subjects: $subjects"
    echo "  Secret verbs: $(echo $has_secrets | tr '\n' ' ')"
  fi
done

Step 3: Check Specific Users/Groups

# Check if a specific user can access secrets:
kubectl auth can-i get secrets --as=developer -n production 2>/dev/null
kubectl auth can-i list secrets --as=developer --all-namespaces 2>/dev/null

# Full permission listing for a user:
kubectl auth can-i --list --as=developer -n secrets-lab 2>/dev/null | grep -i secret

Cleanup

# Remove lab resources
kubectl delete namespace secrets-lab

# Clean up temporary files
rm -f /tmp/api-key.txt /tmp/tls.key /tmp/tls.crt

# OPTIONAL: Remove encryption at rest (only if you want to revert)
# Step 1: Move identity to first position in /etc/kubernetes/enc/enc.yaml
# Step 2: Restart API server
# Step 3: Re-write all secrets: kubectl get secrets -A -o json | kubectl replace -f -
# Step 4: Remove --encryption-provider-config from API server manifest
# Step 5: Delete /etc/kubernetes/enc/

# Restore API server backup if needed:
# sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml.bak \
#   /etc/kubernetes/manifests/kube-apiserver.yaml

Practice Questions

Question 1

You need to enable encryption at rest for Secrets using aescbc. List the complete steps in order.

Show Answer
  1. Generate a 32-byte encryption key: 
    head -c 32 /dev/urandom | base64
  2. Create the EncryptionConfiguration file: ```bash sudo mkdir -p /etc/kubernetes/enc cat «EOF | sudo tee /etc/kubernetes/enc/enc.yaml apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources:
    • resources:
      • secrets providers:
      • aescbc: keys: - name: key1 secret:
      • identity: {} EOF sudo chmod 600 /etc/kubernetes/enc/enc.yaml ```
  3. Configure the API server — edit /etc/kubernetes/manifests/kube-apiserver.yaml:
    • Add flag: --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
    • Add volume mount for /etc/kubernetes/enc
    • Add hostPath volume for /etc/kubernetes/enc
  4. Wait for API server restart (kubelet auto-restarts): 
    kubectl get nodes  # Verify connectivity
  5. Verify new secrets are encrypted: 
    kubectl create secret generic test --from-literal=k=v
etcdctl get /registry/secrets/default/test ... | hexdump -C
# Look for k8s:enc:aescbc:v1:key1
  6. Encrypt all existing secrets: 
    kubectl get secrets -A -o json | kubectl replace -f -

Question 2

Write the kubectl auth can-i commands to verify that ServiceAccount app-sa in namespace myns can ONLY read the db-password Secret and nothing else.

Show Answer 
# Should return "yes" — allowed to get the specific secret:
kubectl auth can-i get secrets/db-password \
  --as=system:serviceaccount:myns:app-sa -n myns
# yes

# Should return "no" — cannot get other secrets:
kubectl auth can-i get secrets/other-secret \
  --as=system:serviceaccount:myns:app-sa -n myns
# no

# Should return "no" — cannot list all secrets:
kubectl auth can-i list secrets \
  --as=system:serviceaccount:myns:app-sa -n myns
# no

# Should return "no" — cannot watch secrets:
kubectl auth can-i watch secrets \
  --as=system:serviceaccount:myns:app-sa -n myns
# no

# Should return "no" — cannot create secrets:
kubectl auth can-i create secrets \
  --as=system:serviceaccount:myns:app-sa -n myns
# no

# Should return "no" — cannot delete secrets:
kubectl auth can-i delete secrets \
  --as=system:serviceaccount:myns:app-sa -n myns
# no

# Should return "no" — cannot access secrets in other namespaces:
kubectl auth can-i get secrets \
  --as=system:serviceaccount:myns:app-sa -n default
# no

Question 3

After enabling encryption at rest, you notice that old Secrets are still in plaintext when read from etcd. What command fixes this?

Show Answer 
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

This command:

  1. Reads every Secret through the API server (which decrypts transparently if needed)
  2. Writes each Secret back via kubectl replace (which encrypts using the current first provider in the EncryptionConfiguration)

After running this, verify with etcdctl:

ETCDCTL_API=3 etcdctl get /registry/secrets/<namespace>/<name> \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C
# Should now show k8s:enc:aescbc:v1:key1 prefix

Why this is needed: Enabling encryption only affects new writes. Existing data stays in its original format until it’s re-written.

Question 4

Your API server won’t start after adding the --encryption-provider-config flag. List the troubleshooting steps.

Show Answer 
# 1. Check API server container logs:
sudo crictl logs $(sudo crictl ps -a | grep kube-apiserver | head -1 | awk '{print $1}') 2>&1 | tail -30

# 2. Common issues and fixes:

# "no such file or directory":
# → Volume mount is wrong or file path doesn't match
sudo ls -la /etc/kubernetes/enc/enc.yaml

# "invalid EncryptionConfiguration":
# → YAML syntax error
python3 -c "import yaml; yaml.safe_load(open('/etc/kubernetes/enc/enc.yaml'))"

# "secret is not of the expected length":
# → The encryption key is not exactly 32 bytes
KEY=$(sudo grep 'secret:' /etc/kubernetes/enc/enc.yaml | awk '{print $2}')
echo -n "$KEY" | base64 -d | wc -c
# Must output: 32

# "unknown apiVersion":
# → Wrong apiVersion in the config
# Must be: apiserver.config.k8s.io/v1

# 3. If stuck, restore the backup:
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml.bak \
  /etc/kubernetes/manifests/kube-apiserver.yaml
# API server will restart with the original config

Question 5

Create a complete RBAC setup (Role + RoleBinding) that allows ServiceAccount monitoring in namespace observability to only read the Secret named grafana-creds. Write the full YAML.

Show Answer 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: grafana-creds-reader
  namespace: observability
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["grafana-creds"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: monitoring-read-grafana-creds
  namespace: observability
subjects:
- kind: ServiceAccount
  name: monitoring
  namespace: observability
roleRef:
  kind: Role
  name: grafana-creds-reader
  apiGroup: rbac.authorization.k8s.io

Apply and verify:

kubectl apply -f rbac.yaml

# Verify:
kubectl auth can-i get secrets/grafana-creds \
  --as=system:serviceaccount:observability:monitoring \
  -n observability
# yes

kubectl auth can-i list secrets \
  --as=system:serviceaccount:observability:monitoring \
  -n observability
# no

kubectl auth can-i get secrets/other-secret \
  --as=system:serviceaccount:observability:monitoring \
  -n observability
# no

Key design decisions:

  • Role (not ClusterRole) — scoped to observability namespace only
  • get only — no list or watch (which would expose all secrets)
  • resourceNames — restricts to grafana-creds only
  • ServiceAccount subject — specific SA, not a group or user
Previous Up Next
Secret Access Control Secrets Management Runtime Security

Updated:

You may also enjoy