CKS — External Secret Stores

10 minute read

External Secret Stores

CKS Exam Domain: Minimize Microservice Vulnerabilities (20%)

Native Kubernetes Secrets work for small clusters, but they have limitations — no centralized management, no built-in rotation, no access audit trail beyond Kubernetes audit logs, and no integration with enterprise secret stores. External secret stores solve these problems by keeping the source of truth outside the cluster while syncing or injecting secrets into Pods at runtime.

1. Why External Stores?

Capability Native K8s Secrets External Stores
Centralized management No — per-cluster Yes — single pane of glass
Automatic rotation No Yes — Vault dynamic secrets, AWS SM rotation
Fine-grained access audit Basic (K8s audit logs) Rich (Vault audit, AWS CloudTrail)
Cross-cluster sharing No — Secrets are per-cluster Yes — single store, multiple clusters
Lease-based access No Yes — time-limited credentials
Compliance / FIPS Depends on etcd encryption KMS-backed, FIPS-validated modules

When to Use External Stores

  • Multi-cluster environments — centralize secrets across dev, staging, production
  • Regulated industries — compliance requires external KMS, HSM, or audit trail
  • Dynamic credentials — database users created on-demand, rotated automatically
  • Zero-trust architectures — Pods get short-lived, scoped credentials

2. HashiCorp Vault Integration

Vault is the most widely used external secret store for Kubernetes. There are two primary integration methods.

Method A: Vault Agent Injector (Sidecar)

The Vault Agent Injector uses a mutating admission webhook that injects a Vault Agent sidecar container into Pods. The sidecar authenticates to Vault and writes secrets to a shared in-memory volume.

┌─────────────────────────────────────────┐
│ Pod                                      │
│                                          │
│  ┌─────────────┐   ┌─────────────────┐  │
│  │ App Container │   │ Vault Agent     │  │
│  │             │   │ (sidecar)       │  │
│  │ reads       │   │                 │  │
│  │ /vault/     │◄──│ writes secrets  │  │
│  │ secrets/    │   │ to shared vol   │  │
│  └─────────────┘   └─────────────────┘  │
│         ▲                   │            │
│         └── shared tmpfs ───┘            │
└─────────────────────────────────────────┘
                    │
                    ▼
           ┌──────────────┐
           │ Vault Server  │
           │ (external)    │
           └──────────────┘

Pod Annotations for Vault Agent Injector

apiVersion: v1
kind: Pod
metadata:
  name: app
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "app-role"
    vault.hashicorp.com/agent-inject-secret-db-password: "secret/data/db/creds"
    vault.hashicorp.com/agent-inject-template-db-password: |
      {{- with secret "secret/data/db/creds" -}}
      {{ .Data.data.password }}
      {{- end -}}
spec:
  serviceAccountName: app-sa
  containers:
  - name: app
    image: myapp:1.0
    # Secret is available at /vault/secrets/db-password

How It Works

  1. Pod is created with Vault annotations
  2. Mutating webhook intercepts the Pod creation
  3. Webhook injects an init container (pre-populates secrets) and a sidecar container (keeps secrets refreshed)
  4. Vault Agent authenticates to Vault using the Pod’s ServiceAccount token (Kubernetes auth method)
  5. Agent retrieves secrets and writes them to /vault/secrets/ (in-memory volume)
  6. Application reads secrets from the filesystem

Method B: Secrets Store CSI Driver with Vault Provider

The CSI driver mounts secrets as a volume directly from Vault, without a sidecar.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
spec:
  provider: vault
  parameters:
    roleName: "app-role"
    vaultAddress: "https://vault.example.com:8200"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/db/creds"
        secretKey: "password"

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  serviceAccountName: app-sa
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: secrets
      mountPath: /mnt/secrets
      readOnly: true
  volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: vault-db-creds

3. AWS Secrets Manager with Secrets Store CSI Driver

For EKS clusters, AWS Secrets Manager integrates via the Secrets Store CSI Driver with the AWS provider.

SecretProviderClass for AWS

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-db-creds
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:us-east-1:123456789012:secret:db-creds-AbCdEf"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: db-username
          - path: password
            objectAlias: db-password

Pod Using AWS Secrets

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  serviceAccountName: app-sa  # Must have IRSA role with SecretsManager access
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: secrets
      mountPath: /mnt/secrets
      readOnly: true
  volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: aws-db-creds

Authentication

  • Uses IAM Roles for Service Accounts (IRSA) — the Pod’s ServiceAccount is associated with an IAM role
  • The IAM role must have secretsmanager:GetSecretValue permission
  • No static credentials are stored in the cluster

4. External Secrets Operator (ESO)

The External Secrets Operator takes a different approach — it syncs secrets from external stores into native Kubernetes Secrets. This means your Pods consume regular K8s Secrets, but the data comes from an external source.

Architecture

┌──────────────┐                    ┌──────────────────┐
│ External      │                    │ Kubernetes        │
│ Secrets       │─── sync ──────────▶│ Secret            │
│ Operator      │                    │ (native K8s)      │
└──────┬───────┘                    └──────────────────┘
       │                                     ▲
       ▼                                     │
┌──────────────┐                    ┌────────┴─────────┐
│ External      │                    │ Pod               │
│ Store         │                    │ (consumes native  │
│ (Vault, AWS,  │                    │  K8s Secret)      │
│  GCP, Azure)  │                    └──────────────────┘
└──────────────┘

ExternalSecret Resource

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
  namespace: production
spec:
  refreshInterval: 1h         # How often to sync from external store
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-creds             # Name of the K8s Secret to create
    creationPolicy: Owner      # ESO owns the Secret lifecycle
  data:
    - secretKey: username      # Key in the K8s Secret
      remoteRef:
        key: secret/data/db/creds
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/db/creds
        property: password

ClusterSecretStore (Vault Example)

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-role"
          serviceAccountRef:
            name: eso-sa
            namespace: external-secrets

Benefits of ESO

  • Pods consume standard K8s Secrets — no sidecar, no CSI driver dependency
  • Works with any application that reads K8s Secrets (no code changes)
  • Supports multiple backends — Vault, AWS, GCP, Azure, 1Password, etc.
  • Built-in refresh interval for rotation
  • ClusterSecretStore allows centralized store config across namespaces

5. Secrets Store CSI Driver — Architecture

The Secrets Store CSI Driver is a CNCF project that provides a common framework for mounting external secrets as volumes.

Components

┌──────────────────────────────────────────────────┐
│ Node                                              │
│                                                    │
│  ┌──────────────────────┐  ┌───────────────────┐  │
│  │ Secrets Store CSI     │  │ Provider Plugin    │  │
│  │ Driver (DaemonSet)    │──│ (Vault/AWS/Azure)  │  │
│  └──────────┬───────────┘  └───────────────────┘  │
│              │                                     │
│              ▼                                     │
│  ┌──────────────────────┐                          │
│  │ Pod                   │                          │
│  │  └── /mnt/secrets/    │ ← secrets mounted here  │
│  └──────────────────────┘                          │
└──────────────────────────────────────────────────┘

SecretProviderClass CRD

The SecretProviderClass is the central CRD that defines:

  • Which provider to use (vault, aws, azure, gcp)
  • Parameters for authentication and secret retrieval
  • Optional: sync to a native K8s Secret (for env var use)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-secrets
  namespace: default
spec:
  provider: vault
  parameters:
    roleName: "my-role"
    vaultAddress: "https://vault:8200"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/myapp"
        secretKey: "password"
  # Optional: also create a K8s Secret from the mounted data
  secretObjects:
    - secretName: db-creds-k8s
      type: Opaque
      data:
        - objectName: db-password
          key: password

6. Comparison Table

Feature Native K8s Secrets Vault Agent Injector Secrets Store CSI External Secrets Operator
Secret source etcd Vault Vault/AWS/Azure/GCP Vault/AWS/Azure/GCP
Extra sidecar No Yes (Vault Agent) No No
CSI driver needed No No Yes No
Operator needed No No Yes (DaemonSet) Yes
Secret type exposed K8s Secret Files in tmpfs Files (+ optional K8s Secret) K8s Secret
Dynamic secrets No Yes Limited Via refresh interval
Rotation Manual Automatic (lease renewal) Via rotation settings Via refreshInterval
Code changes None None (file-based) None (file-based) None
Audit trail K8s audit logs Vault audit logs Vault/cloud audit Vault/cloud audit
CKS exam focus High Conceptual Conceptual Conceptual

CKS Exam Note: The exam primarily tests native Kubernetes Secrets and encryption at rest. External stores like Vault and CSI drivers are tested conceptually — you should understand the architecture, but won’t be asked to install Vault on the exam.

7. Security Best Practices for External Stores

  1. Use Kubernetes auth method — authenticate to Vault/cloud using ServiceAccount tokens, not static credentials
  2. Scope access per namespace — different Vault policies for different namespaces
  3. Use short-lived credentials — Vault dynamic secrets with TTLs, AWS STS temporary credentials
  4. Enable audit logging — both in Kubernetes and in the external store
  5. Network isolation — use NetworkPolicy to restrict which Pods can reach the secret store
  6. Avoid syncing to native Secrets unless necessary — if using CSI driver, mount directly as files
  7. Set refreshInterval — for ESO, configure appropriate sync frequency

Summary

Approach Best For Trade-offs
Native K8s Secrets + encryption at rest Small clusters, CKS exam No rotation, no centralized management
Vault Agent Injector Vault-centric orgs, dynamic secrets Sidecar overhead, complexity
Secrets Store CSI Driver Multi-cloud, file-based secret access DaemonSet required, provider plugins
External Secrets Operator Teams wanting standard K8s Secrets Eventual consistency, sync delay

Practice Questions

Question 1

What is the main advantage of using an external secret store like HashiCorp Vault over native Kubernetes Secrets?

Show Answer

External stores provide capabilities that native Kubernetes Secrets lack:

  1. Centralized management — single source of truth across multiple clusters
  2. Automatic rotation — Vault dynamic secrets issue time-limited credentials
  3. Rich audit trail — Vault audit logs capture every access, not just API server events
  4. Lease-based access — credentials expire automatically, reducing exposure window
  5. Cross-cluster sharing — one Vault instance serves dev, staging, and production

Native Secrets are base64-encoded in etcd, have no built-in rotation, and audit is limited to Kubernetes audit logs.

Question 2

Describe how the Vault Agent Injector works when a Pod with Vault annotations is created.

Show Answer

The Vault Agent Injector uses a mutating admission webhook:

  1. Pod is submitted to the API server with Vault-specific annotations (e.g., vault.hashicorp.com/agent-inject: "true")
  2. The mutating webhook intercepts the Pod creation request
  3. The webhook injects two containers:
    • Init container — pre-populates secrets before the app starts
    • Sidecar container — keeps secrets refreshed throughout the Pod’s lifecycle
  4. A shared in-memory volume (tmpfs) is added for secret files
  5. The Vault Agent authenticates to Vault using the Pod’s ServiceAccount token (Kubernetes auth method)
  6. Agent retrieves secrets and writes them as files to /vault/secrets/
  7. The application container reads secrets from the shared volume

No code changes are needed — the app just reads files from /vault/secrets/.

Question 3

What is a SecretProviderClass and which integration method uses it?

Show Answer

SecretProviderClass is a Custom Resource Definition (CRD) used by the Secrets Store CSI Driver. It defines:

  • Which provider to use (vault, aws, azure, gcp)
  • Parameters for authentication and secret retrieval
  • Optionally, whether to also create a native Kubernetes Secret from the mounted data
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-secrets
spec:
  provider: vault
  parameters:
    roleName: "my-role"
    vaultAddress: "https://vault:8200"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/myapp"
        secretKey: "password"

Pods reference the SecretProviderClass in their volume spec to mount secrets directly from the external store.

Question 4

How does the External Secrets Operator (ESO) differ from the Secrets Store CSI Driver in how Pods consume secrets?

Show Answer

The key difference is in the output format:

  • Secrets Store CSI Driver: Mounts secrets as files directly in the Pod via a CSI volume. The secret data comes from the external store at mount time. Optionally can also create a K8s Secret.
  • External Secrets Operator (ESO): Creates a native Kubernetes Secret by syncing data from the external store. Pods consume the standard K8s Secret using normal env or volumeMount mechanisms.
Aspect CSI Driver ESO
Pod sees Files in a mounted volume Standard K8s Secret
Extra volume needed Yes (CSI volume) No (standard Secret volume)
Works without code changes Yes (read files) Yes (standard K8s Secret)
Requires DaemonSet Yes No (runs as Deployment)
Sync mechanism At Pod start + optional rotation Periodic sync via refreshInterval

ESO is better when you want to keep Pods unaware of the external store. CSI driver is better when you want to avoid creating K8s Secret objects entirely.

Question 5

In a CKS exam scenario, you are asked to ensure that Secrets are not stored in plaintext. You have no access to external tools like Vault. What should you do?

Show Answer

Use encryption at rest with EncryptionConfiguration:

  1. Generate a 32-byte encryption key: 
    head -c 32 /dev/urandom | base64
  2. Create /etc/kubernetes/enc/enc.yaml: ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources:
    • resources:
      • secrets providers:
      • aescbc: keys: - name: key1 secret:
      • identity: {} ```

  3. Add --encryption-provider-config=/etc/kubernetes/enc/enc.yaml to the API server manifest and mount the directory.

  4. Restart the API server and re-encrypt existing Secrets: 
    kubectl get secrets --all-namespaces -o json | kubectl replace -f -

This is the only exam-relevant solution when no external tools are available. The CKS exam tests encryption at rest with aescbc — not external store integration.

Previous Up Next
Encryption at Rest Secrets Management Secret Rotation

Updated:

You may also enjoy