KCSA — Kubernetes Secrets Management

7 minute read

How Kubernetes Secrets Work

A Secret in Kubernetes is a resource designed to hold sensitive data — passwords, tokens, API keys, TLS certificates, and other credentials.

Here’s the critical thing to understand: Kubernetes Secrets are base64-encoded, NOT encrypted by default.

# Create a secret
kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=S3cureP@ss

# View the secret
kubectl get secret db-creds -o yaml

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  username: YWRtaW4=        # base64("admin")
  password: UzNjdXJlUEBzcw==  # base64("S3cureP@ss")

Base64 is encoding, not encryption. Anyone can decode it:

echo "UzNjdXJlUEBzcw==" | base64 --decode
# Output: S3cureP@ss

Without encryption at rest enabled, Secrets are stored in etcd in plaintext (well, base64 — which is essentially plaintext). This is one of the most commonly misunderstood aspects of Kubernetes security.

Secret Types

Kubernetes has several built-in Secret types, each designed for a specific use case.

Type Purpose Auto-Created?
Opaque Generic key-value data. Default type No — user-created
kubernetes.io/tls TLS certificate and private key pair. Requires tls.crt and tls.key fields No — user-created (or cert-manager)
kubernetes.io/dockerconfigjson Docker registry credentials for pulling private images No — user-created via kubectl create secret docker-registry
kubernetes.io/service-account-token Service account bearer token for API server authentication Yes — Kubernetes creates these automatically for service accounts (legacy behavior; bound tokens are preferred since v1.24)
kubernetes.io/basic-auth Username and password for basic authentication No — user-created
kubernetes.io/ssh-auth SSH private key No — user-created
bootstrap.kubernetes.io/token Bootstrap token used during cluster node join Created during kubeadm init/kubeadm join

For the KCSA exam, focus on Opaque (most common), kubernetes.io/tls (used with Ingress), and kubernetes.io/service-account-token (legacy Pod identity).

Consuming Secrets — Environment Variables vs. Volumes

There are two primary ways to expose Secrets to containers. Each has trade-offs.

Environment Variables

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:latest
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-creds
              key: password

Volume Mounts

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: secret-vol
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: secret-vol
      secret:
        secretName: db-creds

Comparison

Aspect Environment Variables Volume Mounts
Access method $DB_PASSWORD in the container process Read from file at /etc/secrets/password
Auto-rotation ❌ No — env vars are set at Pod start and never updated ✅ Yes — kubelet updates the file when the Secret changes (with some delay)
Visibility ⚠️ Visible in kubectl describe pod, process listing (/proc/<pid>/environ), and crash dumps ✅ Only visible inside the container’s filesystem
Container restart needed? Yes — Pod must restart to pick up changes No — files are updated automatically
Ease of use Simple — read from environment Requires file reading logic in the application

Volume mounts are preferred for security-sensitive secrets because they support auto-rotation and are less exposed. Environment variables are acceptable for non-critical configuration.

Encryption at Rest

As discussed in the etcd Security topic, Secrets are stored in plaintext in etcd by default. To encrypt them:

  1. Create an EncryptionConfiguration with a strong provider (aescbc, secretbox, or kms)
  2. Pass it to the API server via --encryption-provider-config
  3. Re-encrypt existing Secrets:
# Re-encrypt all secrets with the new encryption key
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Without encryption at rest, anyone with access to etcd or an etcd backup can read all Secrets. With it, the data in etcd is encrypted and unreadable without the encryption key.

Encryption at rest protects against etcd data theft. It does not protect against someone who has kubectl get secret permissions — RBAC is the control for that.

External Secret Managers

Kubernetes Secrets have limitations — they’re stored in etcd, they require manual rotation, and managing encryption keys is an operational burden. Many organizations use external secret managers to address these gaps.

External Secret Manager Comparison

Solution Provider Key Features
HashiCorp Vault HashiCorp Dynamic secrets, automatic rotation, lease-based access, fine-grained ACLs, audit logging. Supports multiple backends (databases, cloud providers, PKI)
AWS Secrets Manager AWS Automatic rotation, IAM integration, CloudTrail audit, cross-account access. Native to AWS workloads
Azure Key Vault Microsoft Certificate management, HSM-backed keys, Azure AD integration, RBAC. Native to Azure workloads
GCP Secret Manager Google IAM integration, automatic replication, audit logging, versioning. Native to GCP workloads

Integration Patterns

There are two primary ways to integrate external secret managers with Kubernetes:

Pattern Tool How It Works
External Secrets Operator (ESO) external-secrets.io A Kubernetes operator that reads secrets from external providers and creates native Kubernetes Secret objects. Supports all major providers
Secrets Store CSI Driver secrets-store-csi-driver Mounts secrets from external providers directly as volumes in Pods — no Kubernetes Secret object is created. Reduces the attack surface 
External Secrets Operator:
┌──────────────┐     sync      ┌───────────────────┐
│ External       │ ◀──────────▶ │ Kubernetes Secret  │ ──▶ Pod
│ Secret Manager │              │ (auto-created)     │
└──────────────┘              └───────────────────┘

Secrets Store CSI Driver:
┌──────────────┐     mount     ┌──────┐
│ External       │ ◀──────────▶ │ Pod   │  (secret as a volume file)
│ Secret Manager │              │       │
└──────────────┘              └──────┘

The Secrets Store CSI Driver is more secure because the secret never exists as a Kubernetes Secret object — it’s mounted directly from the external provider into the Pod’s filesystem.

RBAC for Secrets

Controlling who can access Secrets via RBAC is one of the most important security configurations in any cluster.

Dangerous RBAC Permissions

Permission Risk
get on secrets Can read the content of specific Secrets
list on secrets Can list all Secrets in a namespace (and read their contents!)
watch on secrets Can watch for Secret changes in real-time
create on secrets Can create new Secrets (may be acceptable)
* (wildcard) on secrets Full access — can read, create, update, delete all Secrets

A common mistake: granting list on resources with a wildcard (resources: ["*"]) in a ClusterRole. This inadvertently grants the ability to list and read all Secrets in every namespace.

Example: Restrictive Role for Secrets

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-creds", "api-key"]  # Only specific secrets
    verbs: ["get"]                           # Only get, not list

Notice:

  • resourceNames restricts access to specific Secrets by name
  • Only get is allowed — not list (which would expose all Secrets in the namespace)
  • The Role is scoped to the production namespace

Common Mistakes

These are the pitfalls the exam tests your ability to identify.

Mistake Why It’s Dangerous Best Practice
Secrets committed to Git Anyone with repo access can read credentials. Git history retains them even after deletion Use external secret managers. If accidentally committed, rotate the secret immediately
Hardcoded secrets in Pod specs Visible in kubectl get pod -o yaml. Stored in etcd alongside the Pod spec Always use Secret references, never inline values
Too-broad RBAC Granting list or * on secrets at cluster scope gives access to every secret in every namespace Use namespace-scoped Roles with resourceNames
No encryption at rest Secrets stored in plaintext in etcd Enable EncryptionConfiguration with aescbc or kms
Environment variables for sensitive data Leaked in logs, crash dumps, /proc filesystem Use volume mounts for sensitive secrets
Not rotating secrets Long-lived credentials increase the blast radius of a breach Use external secret managers with automatic rotation
Service account tokens with excessive permissions Default service account may have more permissions than needed Create dedicated service accounts with minimal RBAC bindings
Secrets in container images Baked into the image layer — anyone who pulls the image gets the secret Never embed secrets in images. Inject at runtime

Immutable Secrets

Since Kubernetes 1.21 (GA), Secrets can be marked as immutable:

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
immutable: true
data:
  password: UzNjdXJlUEBzcw==

Benefits:

  • Protection against accidental updates — prevents modification once created
  • Performance improvement — the kubelet doesn’t need to watch for changes
  • Security — reduces the risk of an attacker modifying a Secret to inject malicious values

To update an immutable Secret, you must delete it and re-create it.

Key Takeaways for the Exam

Concept What to Remember
Base64 ≠ encryption Secrets are base64-encoded, not encrypted. Anyone can decode them
Plaintext in etcd Without encryption at rest, Secrets are stored in plaintext in etcd
Volume mounts > env vars Volumes support auto-rotation and are less exposed than environment variables
Secret types Know Opaque, kubernetes.io/tls, kubernetes.io/service-account-token
Encryption at rest Use EncryptionConfiguration with aescbc or kms provider
External secret managers Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager
ESO vs. CSI Driver ESO creates K8s Secret objects. CSI Driver mounts directly — more secure
RBAC for secrets Never grant list on secrets at cluster scope. Use resourceNames
Never commit to Git Secrets in Git repos are a top real-world vulnerability
Immutable Secrets Prevents accidental or malicious modification. GA since 1.21

Kubernetes Secrets are a convenience, not a vault. For production security, combine encryption at rest, strict RBAC, and external secret managers.

Updated:

You may also enjoy