KCSA — etcd Security

7 minute read

What Is etcd?

etcd is a distributed key-value store that serves as the single source of truth for all Kubernetes cluster state. Every object you create — Pods, Deployments, Services, ConfigMaps, Secrets, RBAC rules — is stored in etcd.

If you can read etcd, you can read everything in the cluster — including every Secret (passwords, API keys, TLS certificates). If you can write to etcd, you can create any object, escalate privileges, or take over the entire cluster.

etcd is arguably the highest-value target in any Kubernetes deployment. Securing it is non-negotiable.

What etcd Stores

Data Type Examples Security Implication
Secrets Passwords, API keys, TLS certificates, tokens Direct credential exposure
ConfigMaps Application configuration, environment variables May contain sensitive connection strings
RBAC Rules Roles, ClusterRoles, Bindings Attacker can read permission structure
Pod Specs Container images, volumes, environment variables May contain inline secrets or privileged settings
Service Accounts Token references, namespace bindings Can be used for lateral movement
Cluster State Node information, endpoints, lease data Full visibility into cluster topology

An attacker with direct access to etcd bypasses the entire API server security pipeline — authentication, authorization, and admission control are all irrelevant.

Encryption at Rest

By default, Kubernetes stores data in etcd in plaintext. This means that if someone gains access to the etcd data directory or a backup, they can read all Secrets in clear text.

Encryption at rest ensures that data is encrypted before it’s written to etcd.

Configuring Encryption at Rest

Encryption at rest is configured via an EncryptionConfiguration resource passed to the API server with the --encryption-provider-config flag.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

The providers list is processed in order:

  • The first provider is used for encryption (writing new data)
  • All listed providers are tried for decryption (reading existing data)
  • The identity: {} provider means “no encryption” — it’s included so that previously unencrypted data can still be read during migration

Encryption Providers

Provider Encryption Strength Key Management Use Case
identity None (plaintext) ❌ None N/A Default. No encryption. Do not use in production
aescbc AES-CBC with PKCS#7 padding ✅ Strong Keys stored in the EncryptionConfiguration file on disk Good for most clusters. Widely used
aesgcm AES-GCM (authenticated encryption) ✅ Strong Keys stored in the EncryptionConfiguration file on disk. Must rotate keys frequently — GCM nonce reuse is dangerous Use only if you have automated key rotation
secretbox XSalsa20 + Poly1305 ✅ Strong Keys stored in the EncryptionConfiguration file on disk Modern, fast, authenticated encryption
kms (v1/v2) Envelope encryption via external KMS ✅✅ Strongest Keys managed by an external Key Management Service (AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault) Recommended for production. Keys never stored on disk

For the KCSA exam, know that kms is the recommended provider for production because the encryption keys are managed externally and never stored on the Kubernetes node’s filesystem.

How Envelope Encryption (KMS) Works

┌──────────────┐    ┌──────────────┐    ┌──────────────────┐
│  API Server   │───▶│  KMS Plugin   │───▶│ External KMS      │
│               │    │               │    │ (AWS KMS, etc.)   │
│ 1. Generates  │    │ 2. Sends DEK  │    │ 3. Encrypts DEK   │
│    a DEK      │    │    to KMS     │    │    with KEK        │
│               │    │               │    │ 4. Returns         │
│ 5. Encrypts   │◀──│               │◀──│    encrypted DEK   │
│    data with  │    │               │    │                    │
│    DEK        │    │               │    │                    │
│ 6. Stores     │    │               │    │                    │
│    encrypted  │    │               │    │                    │
│    data + enc │    │               │    │                    │
│    DEK in etcd│    │               │    │                    │
└──────────────┘    └──────────────┘    └──────────────────┘
  • DEK = Data Encryption Key (generated per encryption operation)
  • KEK = Key Encryption Key (stored in the external KMS, never leaves it)
  • etcd stores the encrypted data plus the encrypted DEK — even if etcd is compromised, the attacker can’t decrypt without access to the KMS

TLS Encryption for etcd Communication

etcd has two types of network communication, and both must be encrypted with TLS:

Peer-to-Peer (etcd member to etcd member)

In a multi-node etcd cluster, members replicate data to each other. This traffic must be encrypted to prevent eavesdropping or tampering.

etcd Flag Purpose
--peer-cert-file TLS certificate for peer communication
--peer-key-file TLS private key for peer communication
--peer-trusted-ca-file CA certificate to verify peer certificates
--peer-client-cert-auth Require client certificate authentication for peers

Client-to-Server (API server to etcd)

The API server is the only component that should communicate with etcd. This connection must be encrypted and authenticated.

etcd Flag Purpose
--cert-file TLS certificate for client-facing communication
--key-file TLS private key for client-facing communication
--trusted-ca-file CA certificate to verify client certificates
--client-cert-auth Require client certificate authentication

On the API server side, corresponding flags configure the client connection:

API Server Flag Purpose
--etcd-certfile Client certificate for connecting to etcd
--etcd-keyfile Client private key for connecting to etcd
--etcd-cafile CA certificate to verify etcd’s server certificate

Never run etcd without TLS in any environment — not even in development. An unencrypted etcd connection means anyone on the network can read all cluster secrets.

Access Control

Restrict Access to the API Server Only

The only Kubernetes component that should communicate with etcd is the API server. No other component — not kubelets, not controllers, not users — should have direct access.

Control How to Implement
Network isolation Place etcd on a dedicated network or subnet. Use firewall rules to allow traffic only from the API server’s IP(s)
Client certificate auth Configure --client-cert-auth=true on etcd. Only the API server’s client certificate is trusted
Dedicated etcd nodes In production, run etcd on dedicated machines (not on control plane nodes that also run the API server) for stronger isolation
No anonymous access etcd should never accept unauthenticated connections

File System Permissions

etcd stores data in a local directory (default: /var/lib/etcd). This directory must be protected:

  • Ownership: etcd:etcd (or the user running the etcd process)
  • Permissions: 700 (only the owner can read/write/execute)
  • The etcd data directory contains the full cluster state in plaintext (unless encryption at rest is enabled)

Backup Security

Regular etcd backups are essential for disaster recovery — but backups contain the same sensitive data as the live etcd store.

etcd Backup Command

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.crt \
  --cert=/etc/etcd/server.crt \
  --key=/etc/etcd/server.key

Securing Backups

Practice Description
Encrypt backups Encrypt the snapshot file before storing it (e.g., using GPG or envelope encryption)
Secure storage Store backups in an encrypted, access-controlled location (e.g., S3 with SSE-KMS, encrypted disk)
Restrict access Only authorized personnel should be able to restore backups — a restore replaces the entire cluster state
Test restores Regularly test backup restores to a separate environment to verify integrity
Rotate backups Keep only the necessary number of backups and securely delete old ones

An unencrypted etcd backup is a goldmine for attackers. It contains every Secret, every RBAC rule, every ConfigMap — the entire cluster in a single file.

Network Isolation for etcd

etcd should be as isolated as possible from the rest of the network:

  • Dedicated subnet — etcd nodes should be on a separate network segment from worker nodes
  • Firewall rules — allow inbound traffic only from:
    • API server (client-to-server on port 2379)
    • Other etcd members (peer-to-peer on port 2380)
  • No public access — etcd should never be reachable from the internet
  • No access from worker nodes — a compromised workload on a worker node should not be able to reach etcd directly

In managed Kubernetes services (EKS, GKE, AKS), the cloud provider manages etcd and its network isolation. You don’t have direct access, which is actually a security benefit.

Key Takeaways for the Exam

Concept What to Remember
etcd stores everything All cluster state — Secrets, ConfigMaps, RBAC rules, Pod specs — is in etcd
Default is plaintext Without encryption at rest, Secrets are stored in plaintext in etcd
Encryption providers identity (none), aescbc, aesgcm, secretbox, kms — know what each does
KMS is recommended Envelope encryption via an external KMS keeps keys off the node. Best practice for production
TLS is mandatory Both peer-to-peer and client-to-server etcd communication must use TLS
API server only Only the API server should have access to etcd — restrict via network and certificates
Backups are sensitive An etcd backup contains all cluster secrets — encrypt and secure it
Direct access = full compromise Anyone who can read/write etcd directly bypasses all Kubernetes security controls

etcd security is foundational. If etcd is compromised, nothing else in the cluster matters. Protect it like the crown jewels of your cluster.

Updated:

You may also enjoy