CKS — Encryption at Rest

11 minute read

Encryption at Rest

CKS Exam Domain: Minimize Microservice Vulnerabilities (20%)

THIS IS A MUST-KNOW CKS TOPIC. Encryption at rest is one of the most frequently tested topics on the CKS exam. You must be able to create an EncryptionConfiguration, configure the API server, and verify that Secrets are encrypted in etcd.

By default, Kubernetes stores Secrets in etcd as base64-encoded plaintext. Encryption at rest ensures that even if an attacker gains access to etcd data or backups, they cannot read Secret values without the encryption key.

1. The Problem — Secrets in etcd Without Encryption

# Without encryption at rest, Secrets are readable in etcd:
ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-creds \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

# You will see the base64-encoded values in PLAINTEXT in the hex dump
# Example: "password123" is visible as ASCII text

This means:

  • etcd backups contain all secrets in plaintext
  • etcd disk contains all secrets in plaintext
  • Anyone with etcd TLS credentials can dump every credential in your cluster

2. EncryptionConfiguration Resource

Kubernetes uses the EncryptionConfiguration resource to define how Secrets (and other resources) are encrypted before being written to etcd.

Complete EncryptionConfiguration YAML

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64_ENCODED_32_BYTE_KEY>
      - identity: {}

Key Points

  • resources specifies which Kubernetes resources to encrypt (most commonly secrets)
  • providers are tried in order — the first provider is used for writing (encryption)
  • All providers are tried for reading (decryption) — this allows key rotation
  • identity: {} at the end means “read unencrypted data” — needed to read Secrets created before encryption was enabled
  • Order matters: If identity is first, nothing gets encrypted

Full Example with Multiple Resources

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: dGhpcyBpcyBhIDMyIGJ5dGUga2V5IGZvciBhZXNjYmM=
      - identity: {}

3. Encryption Providers

Kubernetes supports multiple encryption providers. Choose based on your security requirements and environment.

Provider Encryption Strength Key Management CKS Exam
identity None No encryption — plaintext N/A Default (insecure)
aescbc AES-CBC with PKCS#7 padding Strong Static key in config Recommended for exam
aesgcm AES-GCM Strong + authenticated Static key, must rotate frequently Not recommended (nonce reuse risk)
secretbox XSalsa20 + Poly1305 Strong + authenticated Static key in config Good alternative
kms v1 Envelope encryption Strongest External KMS (Vault, AWS KMS, etc.) Production-grade
kms v2 Envelope encryption (improved) Strongest External KMS with DEK reuse Production-grade

Why aescbc for the Exam

  • Simplest working encryption provider
  • No external dependencies (no KMS server required)
  • All configuration is self-contained in a single file
  • Well-documented in Kubernetes docs
  • Most likely what the exam environment supports

Exam Tip: Unless the question specifically asks for KMS, use aescbc. It’s fast to configure and easy to verify.

Provider Comparison — Encryption Flow

identity (no encryption):
  Secret → base64 → etcd (plaintext)

aescbc:
  Secret → AES-CBC encrypt with static key → etcd (encrypted blob)

kms v2 (envelope encryption):
  Secret → Encrypt with DEK → Store encrypted DEK + encrypted secret in etcd
           └── DEK encrypted by remote KMS (Vault, AWS KMS, etc.)

4. Generating the Encryption Key

The encryption key must be 32 bytes for aescbc, base64-encoded:

# Generate a 32-byte random key and base64-encode it:
head -c 32 /dev/urandom | base64
# Example output: dGhpcyBpcyBhIDMyIGJ5dGUga2V5IGZvciBhZXNjYmM=

Important: This key is the only thing protecting your Secrets. If it’s compromised, all encrypted Secrets can be decrypted. Store it securely and restrict file permissions.

5. Step-by-Step: Enable Encryption at Rest

Step 1: Generate the Encryption Key

ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
echo $ENCRYPTION_KEY

Step 2: Create the EncryptionConfiguration File

sudo mkdir -p /etc/kubernetes/enc

cat <<EOF | sudo tee /etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

Step 3: Set Restrictive Permissions

sudo chmod 600 /etc/kubernetes/enc/enc.yaml
sudo chown root:root /etc/kubernetes/enc/enc.yaml

Step 4: Configure the API Server

Edit the kube-apiserver static Pod manifest:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add the flag to the command:

spec:
  containers:
  - command:
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml   # ← ADD THIS
    # ... other flags ...

Add the volume and volume mount:

spec:
  containers:
  - name: kube-apiserver
    volumeMounts:
    - name: enc-config
      mountPath: /etc/kubernetes/enc
      readOnly: true
    # ... other volume mounts ...
  volumes:
  - name: enc-config
    hostPath:
      path: /etc/kubernetes/enc
      type: DirectoryOrCreate
  # ... other volumes ...

Step 5: Wait for API Server Restart

The kubelet detects the manifest change and automatically restarts the API server Pod:

# Watch for the API server to come back:
watch -n 1 "kubectl get pods -n kube-system | grep kube-apiserver"

# Or check container status:
sudo crictl ps | grep kube-apiserver

# Give it 30-60 seconds. If it doesn't come back, check logs:
sudo crictl logs $(sudo crictl ps -a | grep kube-apiserver | head -1 | awk '{print $1}')

Step 6: Verify — Create a New Secret and Check etcd

# Create a test secret:
kubectl create secret generic test-encryption \
  --from-literal=mykey=mydata

# Read it from etcd:
ETCDCTL_API=3 etcdctl get /registry/secrets/default/test-encryption \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

Before encryption: You’ll see mydata or its base64 encoding in ASCII in the hex dump.

After encryption: You’ll see an encrypted blob. The data starts with the prefix:

k8s:enc:aescbc:v1:key1:<encrypted binary data>

6. Encrypting Existing Secrets

Enabling encryption at rest only affects newly created or updated Secrets. Existing Secrets remain unencrypted in etcd until they are re-written.

Encrypt All Existing Secrets

# This reads every Secret and writes it back, triggering encryption:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Verify Encryption for a Specific Secret

# Check an existing secret in etcd:
ETCDCTL_API=3 etcdctl get /registry/secrets/kube-system/default-token-xxxxx \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

# Look for the k8s:enc:aescbc:v1:key1 prefix
# If you see plaintext base64, the secret hasn't been re-encrypted yet

7. Verifying Encryption in etcd — Detailed

This is a critical exam skill — knowing how to verify that encryption is actually working.

Before Encryption (identity provider)

ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Output will contain readable content like:
# k8s
#
# v1Secret...
# password
# s3cur3P@ss    ← Secret value visible!

After Encryption (aescbc provider)

ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

# Output will show:
# 00000000  ... k8s:enc:aescbc:v1:key1 ...
# followed by binary encrypted data
# The actual secret value is NOT visible

Key Prefixes by Provider

Provider etcd Value Prefix
identity k8s\n\x00 (no prefix — plaintext)
aescbc k8s:enc:aescbc:v1:<key-name>:
aesgcm k8s:enc:aesgcm:v1:<key-name>:
secretbox k8s:enc:secretbox:v1:<key-name>:
kms v1 k8s:enc:kms:v1:<config-name>:
kms v2 k8s:enc:kms:v2:<config-name>:

8. Key Rotation

To rotate encryption keys without downtime:

Step 1: Add the New Key (First Position) and Keep the Old Key

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key2              # ← New key (used for writing)
              secret: <NEW_KEY_BASE64>
            - name: key1              # ← Old key (still used for reading)
              secret: <OLD_KEY_BASE64>
      - identity: {}

Step 2: Restart API Server

# The kubelet auto-restarts after manifest change
# Wait for the API server to come back:
kubectl get nodes  # Simple connectivity test

Step 3: Re-encrypt All Secrets

# Re-encrypt with the new key:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Step 4: Remove the Old Key

Once all Secrets are re-encrypted, remove the old key from the config:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key2
              secret: <NEW_KEY_BASE64>
      - identity: {}

Step 5: Restart API Server Again

# Confirm all secrets are still readable:
kubectl get secrets --all-namespaces

9. Decrypting — Reverting to No Encryption

To disable encryption and revert to plaintext storage:

Step 1: Move identity to First Position

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - identity: {}        # ← Now first — new writes are plaintext
      - aescbc:
          keys:
            - name: key1
              secret: <KEY_BASE64>   # ← Still needed to read encrypted secrets

Step 2: Restart API Server and Re-write All Secrets

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Step 3: Remove the EncryptionConfiguration Entirely

Remove the --encryption-provider-config flag from the API server manifest and delete the config file.

10. Troubleshooting

API Server Won’t Start After Adding Encryption Config

# Check API server container logs:
sudo crictl logs $(sudo crictl ps -a | grep kube-apiserver | head -1 | awk '{print $1}')

# Common errors:
# - "no such file or directory" → Volume mount path is wrong
# - "invalid EncryptionConfiguration" → YAML syntax error
# - "secret is not of the expected length" → Key is not 32 bytes (base64 of 32 bytes)

Secrets Cannot Be Read After Enabling Encryption

This happens when identity: {} is missing from the providers list and there are pre-existing unencrypted Secrets:

# WRONG — cannot read old unencrypted secrets:
providers:
  - aescbc:
      keys:
        - name: key1
          secret: <KEY>

# CORRECT — can read both encrypted and unencrypted:
providers:
  - aescbc:
      keys:
        - name: key1
          secret: <KEY>
  - identity: {}           # ← Needed for reading old unencrypted data

Verifying the Config Is Loaded

# Check if the API server flag is present:
ps aux | grep kube-apiserver | grep encryption-provider-config

# Check if the config file is mounted:
sudo crictl inspect $(sudo crictl ps | grep kube-apiserver | awk '{print $1}') | \
  jq '.info.config.mounts[] | select(.destination | contains("enc"))'

Summary

Step Action Command/File
1 Generate key head -c 32 /dev/urandom \| base64
2 Create config /etc/kubernetes/enc/enc.yaml
3 Configure API server --encryption-provider-config flag + volume mount
4 Restart API server Edit static Pod manifest, kubelet auto-restarts
5 Verify new secrets etcdctl get ... \| hexdump -C — look for k8s:enc:aescbc
6 Encrypt existing kubectl get secrets --all-namespaces -o json \| kubectl replace -f -
7 Key rotation Add new key first, re-encrypt, remove old key

Practice Questions

Question 1

Write a complete EncryptionConfiguration that encrypts Secrets using aescbc with a key named prod-key-1.

Show Answer 
# First, generate a 32-byte key:
head -c 32 /dev/urandom | base64
# Example: czNjdXIzUEBzc3dvcmQxMjM0NTY3ODkwMTIzNDU2Nzg=

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: prod-key-1
              secret: czNjdXIzUEBzc3dvcmQxMjM0NTY3ODkwMTIzNDU2Nzg=
      - identity: {}

Key points:

  • apiVersion: apiserver.config.k8s.io/v1 (must be exact)
  • aescbc is listed before identity (otherwise nothing gets encrypted)
  • identity: {} at the end allows reading pre-existing unencrypted Secrets
  • The key must be exactly 32 bytes, base64-encoded

Question 2

After enabling encryption at rest, you create a new Secret. How do you verify it’s actually encrypted in etcd?

Show Answer

Use etcdctl to read the Secret directly from etcd and inspect the raw data:

# Create a test secret:
kubectl create secret generic verify-enc --from-literal=test=hello

# Read from etcd:
ETCDCTL_API=3 etcdctl get /registry/secrets/default/verify-enc \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

Look for the prefix k8s:enc:aescbc:v1:<key-name> in the output. If encryption is working:

  • The data will be binary (not human-readable)
  • The prefix k8s:enc:aescbc:v1:key1 will be visible

If encryption is NOT working:

  • You’ll see the Secret data in plaintext or base64
# Cleanup:
kubectl delete secret verify-enc

Question 3

You enabled encryption at rest, but Secrets created before the change are still unencrypted in etcd. How do you fix this?

Show Answer

Enabling encryption only affects new writes. Existing Secrets must be re-written to trigger encryption:

# Re-encrypt all existing Secrets:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

This command:

  1. Reads every Secret via the API server (which decrypts them transparently)
  2. Writes them back via kubectl replace (which re-encrypts with the current provider)

After this, verify a pre-existing Secret:

ETCDCTL_API=3 etcdctl get /registry/secrets/kube-system/<secret-name> \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C
# Should now show k8s:enc:aescbc:v1:key1 prefix

Question 4

What happens if you put identity: {} as the first provider in your EncryptionConfiguration?

Show Answer

If identity: {} is the first provider, no encryption occurs. The first provider in the list is used for writing, and identity means “no encryption.”

# WRONG — this does NOT encrypt anything:
providers:
  - identity: {}              # ← First = used for writing = NO encryption
  - aescbc:
      keys:
        - name: key1
          secret: <KEY>

# CORRECT — aescbc first = encryption active:
providers:
  - aescbc:
      keys:
        - name: key1
          secret: <KEY>
  - identity: {}             # ← Only used as fallback for reading old data

The identity provider is useful as the last provider to allow reading Secrets that were stored before encryption was enabled. But it must never be the first provider if you want encryption.

Question 5

You need to rotate the encryption key from key1 to key2 without downtime. Describe the steps.

Show Answer

Step 1: Add the new key before the old key in the EncryptionConfiguration:

providers:
  - aescbc:
      keys:
        - name: key2              # New key — used for writing
          secret: <NEW_KEY>
        - name: key1              # Old key — used for reading
          secret: <OLD_KEY>
  - identity: {}

Step 2: Restart the API server (edit static Pod manifest, kubelet auto-restarts):

# Wait for API server:
kubectl get nodes

Step 3: Re-encrypt all Secrets with the new key:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Step 4: Remove the old key from the configuration:

providers:
  - aescbc:
      keys:
        - name: key2
          secret: <NEW_KEY>
  - identity: {}

Step 5: Restart the API server again. All Secrets are now encrypted with key2 only.

This process ensures zero downtime because both keys are available for reading during the transition.

Previous Up Next
Kubernetes Secrets Deep Dive Secrets Management External Secret Stores

Updated:

You may also enjoy