CKS — mTLS & Pod-to-Pod Encryption

mTLS & Pod-to-Pod Encryption

By default, all traffic between Pods in a Kubernetes cluster is unencrypted. Any compromised Pod can sniff network traffic on the same node, capturing credentials, API tokens, and sensitive data in transit. Mutual TLS (mTLS) encrypts east-west traffic and authenticates both sides of every connection, preventing eavesdropping and lateral movement.

---

1. Why Encrypt East-West Traffic?

The Problem: Unencrypted Pod Traffic

Default Kubernetes Networking:

  Pod A ──── plaintext HTTP ────→ Pod B
       │                           │
       │    Attacker Pod (same node or network)
       │         │
       │         └── tcpdump / packet capture
       │              → Sees all data in cleartext:
       │                - API tokens
       │                - Database credentials
       │                - User PII
       │                - Session cookies

With mTLS

mTLS-Protected Networking:

  Pod A ──── encrypted TLS ────→ Pod B
  (client cert)              (server cert)
       │                           │
       │    Attacker Pod
       │         │
       │         └── tcpdump → Encrypted gibberish
       │              Cannot decrypt without certs
       │              Cannot impersonate either Pod

What mTLS Provides

Property	Description
Encryption	All traffic encrypted with TLS — packet sniffing sees only ciphertext
Authentication	Both client and server present certificates — no impersonation
Identity	Certificates tied to service identity (SPIFFE ID) — “is this really Pod B?”
Integrity	TLS ensures data cannot be tampered with in transit

Threat Model: Lateral Movement

Attack Chain Without mTLS:
  1. Attacker compromises Pod A (e.g., RCE vulnerability)
  2. From Pod A, sniff network traffic → capture DB password
  3. Connect to database → exfiltrate all data
  4. Impersonate Pod B → access additional services

Attack Chain With mTLS:
  1. Attacker compromises Pod A
  2. Sniff traffic → ENCRYPTED (useless)
  3. Try to connect to database → NO VALID CLIENT CERTIFICATE → rejected
  4. Impersonate Pod B → NO MATCHING CERTIFICATE → rejected
  5. Attack stops at Pod A

---

2. Service Mesh Options

Service meshes implement mTLS transparently by injecting sidecar proxies into every Pod. The application doesn’t need to handle TLS — the sidecar does it.

How a Service Mesh Implements mTLS

Without Service Mesh:
  App A → plaintext → App B

With Service Mesh (Sidecar Pattern):
  App A → plaintext → Envoy Proxy A → TLS → Envoy Proxy B → plaintext → App B
  (localhost)         (sidecar)              (sidecar)        (localhost)

The application communicates over localhost (plaintext) with its own sidecar. The sidecar handles TLS encryption, certificate rotation, and mutual authentication. The application code never changes.

Istio vs Linkerd Comparison

Feature	Istio	Linkerd
Proxy	Envoy (C++)	linkerd2-proxy (Rust)
Resource overhead	Higher (~50MB per sidecar)	Lower (~20MB per sidecar)
Complexity	Higher — many configuration options	Lower — simpler by design
mTLS	Automatic, per-namespace/workload control	Automatic, enabled by default
Identity	SPIFFE-based (X.509 certificates)	SPIFFE-based (X.509 certificates)
Certificate authority	istiod (built-in) or external CA	identity controller (built-in)
Policy control	PeerAuthentication, AuthorizationPolicy	Server, ServerAuthorization
Observability	Kiali, Prometheus, Jaeger	Linkerd Viz, Prometheus, Jaeger
CKS relevance	More likely on the exam	Less common on the exam

Exam Tip: The CKS exam is more likely to reference Istio concepts. Focus on PeerAuthentication modes (STRICT, PERMISSIVE, DISABLE) and how to verify mTLS is active.

---

3. Automatic mTLS with Istio

Sidecar Injection

Istio injects an Envoy sidecar proxy into every Pod in labeled namespaces:

# Enable automatic sidecar injection for a namespace
kubectl label namespace production istio-injection=enabled

# All new Pods in this namespace get an Envoy sidecar
kubectl get pods -n production
# NAME                     READY   STATUS
# web-app-abc123           2/2     Running    ← 2 containers (app + sidecar)
# api-server-def456        2/2     Running

How Istio mTLS Works

Certificate Lifecycle:
  1. istiod generates a root CA certificate
  2. Each Envoy sidecar requests a certificate from istiod
  3. istiod issues a short-lived X.509 cert with SPIFFE identity
     (e.g., spiffe://cluster.local/ns/production/sa/web-app)
  4. Sidecars automatically rotate certificates before expiry

Connection Flow:
  Pod A (Envoy) ──── TLS handshake ────→ Pod B (Envoy)
  • Client presents certificate (identity: Pod A's service account)
  • Server presents certificate (identity: Pod B's service account)
  • Both verify via shared root CA
  • Encrypted channel established
  • Application data flows through encrypted tunnel

---

4. PeerAuthentication Resource

PeerAuthentication controls the mTLS policy for workloads in Istio. It defines whether mTLS is required, optional, or disabled.

Modes

Mode	Behavior	Use Case
STRICT	Only mTLS connections accepted — plaintext rejected	Production namespaces (secure default)
PERMISSIVE	Accepts both mTLS and plaintext connections	Migration phase — allows non-mesh services to communicate
DISABLE	No mTLS — all traffic is plaintext	Legacy workloads that cannot support TLS (temporary)
UNSET	Inherits from parent (namespace → mesh-wide)	Default behavior

Mesh-Wide STRICT mTLS

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system       # Mesh-wide when in istio-system
spec:
  mtls:
    mode: STRICT

This enforces mTLS for all services across the entire mesh. Any plaintext connection is rejected.

Namespace-Level PeerAuthentication

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production         # Applies to all Pods in this namespace
spec:
  mtls:
    mode: STRICT

Workload-Specific PeerAuthentication

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: db-strict
  namespace: production
spec:
  selector:
    matchLabels:
      app: database             # Only applies to Pods with this label
  mtls:
    mode: STRICT

Per-Port Configuration

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: web-app-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: web
  mtls:
    mode: STRICT
  portLevelMtls:
    8080:
      mode: PERMISSIVE          # Port 8080 accepts plaintext (health checks)
    443:
      mode: STRICT              # Port 443 requires mTLS

---

5. DestinationRule for mTLS

While PeerAuthentication controls the server side (what the destination accepts), DestinationRule controls the client side (what the source sends).

Enable mTLS on Outgoing Traffic

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: database-mtls
  namespace: production
spec:
  host: database.production.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL        # Use Istio's auto-generated certificates

TLS Modes in DestinationRule

Mode	Behavior
DISABLE	Send plaintext
SIMPLE	One-way TLS (client verifies server, no client cert)
MUTUAL	mTLS with custom certificates (you provide certs)
ISTIO_MUTUAL	mTLS using Istio’s auto-managed certificates (recommended)

Mesh-Wide DestinationRule

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: mesh-wide-mtls
  namespace: istio-system
spec:
  host: "*.local"               # All services in the mesh
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

---

6. Verifying mTLS Is Active

Using istioctl

# Check mTLS status for all services
istioctl x describe service web-app.production

# Check if a specific connection uses mTLS
istioctl authn tls-check web-app-pod.production database.production.svc.cluster.local
# HOST:PORT                                    STATUS    SERVER     CLIENT
# database.production.svc.cluster.local:3306   OK        STRICT     ISTIO_MUTUAL

# Check proxy configuration
istioctl proxy-config listeners web-app-pod -n production

Using Kiali Dashboard

Kiali is Istio’s observability dashboard. It shows:

• Which services have mTLS enabled (lock icon)
• Traffic flows between services
• mTLS misconfigurations

# Access Kiali
istioctl dashboard kiali

Using tcpdump (Verify Encryption)

# Deploy a debug Pod on the same node
kubectl debug node/node01 -it --image=nicolaka/netshoot

# Capture traffic between Pods
tcpdump -i eth0 -nn port 8080 -w /tmp/capture.pcap

# If mTLS is working, the captured traffic is encrypted
# You'll see TLS handshake packets (ClientHello, ServerHello)
# But no readable application data

Checking Certificates

# View the certificate issued to a sidecar
istioctl proxy-config secret web-app-pod -n production -o json | \
  jq '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | \
  tr -d '"' | base64 -d | openssl x509 -text -noout

# Look for:
# Subject: O = cluster.local (trust domain)
# URI: spiffe://cluster.local/ns/production/sa/web-app
# Validity: Not After (short-lived, auto-rotated)

---

7. mTLS vs NetworkPolicies

mTLS and NetworkPolicies are complementary, not replacements for each other. They operate at different layers and protect against different threats.

Feature	NetworkPolicies	mTLS
Layer	L3/L4 (IP, port)	L7 (application, identity)
Controls	Which Pods can communicate	How Pods communicate (encrypted, authenticated)
Identity	IP-based (labels → IPs)	Certificate-based (SPIFFE identity)
Encryption	No	Yes
Authentication	No	Yes (mutual)
Blocks traffic	Yes (deny by default)	Yes (reject without valid cert in STRICT mode)
Protects against sniffing	No	Yes
Protects against impersonation	Partially (IP spoofing possible)	Yes (certificate-based identity)

Defense in Depth: Use Both

# Layer 1: NetworkPolicy — restrict WHO can talk to the database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: database-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes: ["Ingress"]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-server
    ports:
    - port: 5432
---
# Layer 2: PeerAuthentication — encrypt and authenticate the connection
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: database-mtls
  namespace: production
spec:
  selector:
    matchLabels:
      app: database
  mtls:
    mode: STRICT

NetworkPolicy blocks unauthorized Pods from connecting. mTLS ensures that even authorized connections are encrypted and both parties are authenticated.

---

8. Linkerd Automatic mTLS

Linkerd enables mTLS by default with zero configuration — every meshed connection is automatically encrypted and authenticated.

Enabling Linkerd mTLS

# Install Linkerd
linkerd install | kubectl apply -f -

# Inject sidecar into a namespace
kubectl annotate namespace production linkerd.io/inject=enabled

# Restart Deployments to trigger injection
kubectl rollout restart deployment -n production

Verifying mTLS in Linkerd

# Check mTLS stats
linkerd viz stat deploy -n production
# NAME         MESHED   SUCCESS   RPS   LATENCY_P50   LATENCY_P99
# web-app       1/1     100.00%   10    5ms           25ms
# api-server    1/1     100.00%    8    3ms           15ms

# Check if connections are secured
linkerd viz edges deploy -n production
# SRC            DST            SRC_NS       DST_NS       SECURED
# web-app        api-server     production   production   √
# api-server     database       production   production   √

Server Resource (Linkerd’s Policy)

apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  name: database-server
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  port: 5432
  proxyProtocol: TLS           # Require mTLS

---

9. Gradual mTLS Rollout Strategy

Phase 1: PERMISSIVE Mode (Discovery)

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: PERMISSIVE            # Accept both mTLS and plaintext

• Services in the mesh use mTLS automatically
• Non-mesh services can still communicate via plaintext
• Monitor with Kiali to identify services not yet meshed

Phase 2: Identify Non-Mesh Services

# Find services still sending plaintext
istioctl x describe service web-app.production
# Look for: "Clients using plaintext"

# Check which Pods don't have sidecars
kubectl get pods -n production -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.name}{","}{end}{"\n"}{end}' | grep -v istio-proxy

Phase 3: STRICT Mode (Enforcement)

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT                # Reject anything without a valid mTLS connection

After switching to STRICT, any service without a sidecar will be unable to communicate with meshed services.

---

Summary

Component	Detail
Why mTLS	Encrypts east-west traffic, prevents sniffing and lateral movement
Service mesh	Transparent mTLS via sidecar proxies (Envoy/linkerd2-proxy)
Istio PeerAuthentication	STRICT (mTLS only), PERMISSIVE (both), DISABLE (plaintext)
DestinationRule	Client-side TLS config — ISTIO_MUTUAL for auto-managed certs
Verification	istioctl, Kiali, tcpdump, certificate inspection
Linkerd	mTLS enabled by default — zero-config encryption
vs NetworkPolicies	Complementary — NetworkPolicies control access, mTLS encrypts and authenticates
Rollout	PERMISSIVE → identify non-mesh services → STRICT

---

Practice Questions

Q1: By default, is traffic between Pods in a Kubernetes cluster encrypted? What threat does this create?

No, by default all Pod-to-Pod traffic is unencrypted plaintext.

Kubernetes networking (CNI plugins like Calico, Flannel, Cilium) does not encrypt traffic by default. Packets between Pods travel as plaintext across the cluster network.

Threats:

1. Eavesdropping — Any compromised Pod on the same node (or with access to the network) can capture traffic using tools like tcpdump. This exposes:
   • Database credentials in connection strings
   • API tokens in HTTP headers
   • User PII in request/response bodies
   • Session cookies
2. Lateral movement — After capturing credentials from sniffed traffic, an attacker can:
   • Connect to databases with stolen passwords
   • Call internal APIs with stolen tokens
   • Impersonate legitimate services
3. Man-in-the-middle — An attacker can intercept and modify traffic between services

Solution: mTLS encrypts all Pod-to-Pod traffic and authenticates both endpoints, preventing all three threats.

Q2: Write a PeerAuthentication resource that enforces STRICT mTLS for all services in the production namespace, except port 8081 on the web-app which should allow plaintext (for health checks from a non-mesh load balancer).

You need two resources — a namespace-wide default and a workload-specific override:

# Namespace-wide: STRICT mTLS for everything in production
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT
---
# Workload-specific: allow plaintext on port 8081 for web-app
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: web-app-health
  namespace: production
spec:
  selector:
    matchLabels:
      app: web-app
  mtls:
    mode: STRICT
  portLevelMtls:
    8081:
      mode: PERMISSIVE        # Health check port accepts plaintext

How this works:

• All services in production require mTLS (STRICT)
• web-app requires mTLS on all ports except 8081
• Port 8081 accepts both mTLS and plaintext (PERMISSIVE) for the load balancer health check
• The workload-specific policy’s portLevelMtls overrides the general mode for that port only

Q3: What is the difference between PeerAuthentication and DestinationRule in the context of mTLS?

Aspect	PeerAuthentication	DestinationRule
Controls	Server-side (receiving end)	Client-side (sending end)
Question it answers	“What does the destination accept?”	“What does the source send?”
mTLS modes	STRICT, PERMISSIVE, DISABLE	DISABLE, SIMPLE, MUTUAL, ISTIO_MUTUAL
Scope	Mesh-wide, namespace, or workload	Per-host traffic policy
Effect	Determines if plaintext is rejected	Determines if TLS is initiated

How they work together:

PeerAuthentication (server side):
  "I only accept mTLS connections" (STRICT mode)

DestinationRule (client side):
  "I will send mTLS connections to this service" (ISTIO_MUTUAL)

Both must agree:
  Client sends mTLS + Server expects mTLS = ✓ Connection works
  Client sends plaintext + Server expects mTLS = ✗ Connection rejected
  Client sends mTLS + Server expects plaintext = ✗ Handshake fails

In practice, with automatic mTLS in Istio, you primarily configure PeerAuthentication. Istio’s control plane automatically configures the client side. You only need explicit DestinationRule for non-standard TLS configurations or when communicating with external services.

Q4: Explain why mTLS and NetworkPolicies are complementary, not replacements. Give a scenario where having only one is insufficient.

Scenario 1: NetworkPolicies only (no mTLS)

Setup: NetworkPolicy allows Pod-A → Pod-B on port 5432

Attack: Attacker compromises Pod-C on the same node
  1. Pod-C runs tcpdump → captures Pod-A's database credentials (plaintext)
  2. Attacker creates a new legitimate connection from Pod-A
     (if they can inject code) using stolen credentials
  3. NetworkPolicy allows it (Pod-A is whitelisted)
  4. Data exfiltrated

NetworkPolicy controlled access but didn’t prevent eavesdropping. mTLS would have encrypted the traffic, making the captured packets useless.

Scenario 2: mTLS only (no NetworkPolicies)

Setup: mTLS STRICT mode on all services

Attack: Attacker compromises Pod-C (a meshed Pod with a valid sidecar cert)
  1. Pod-C has a valid mTLS certificate (auto-issued by the mesh)
  2. Attacker uses Pod-C to connect to the database
  3. mTLS accepts the connection (valid certificate from the mesh)
  4. Data exfiltrated

mTLS encrypts and authenticates, but any meshed Pod with a certificate can connect to any other meshed service. There’s no access control at the network layer.

Defense in depth — use both:

• NetworkPolicies → control which Pods can communicate (deny by default, allow specific traffic)
• mTLS → encrypt the allowed traffic and authenticate identities
• Together → attacker must compromise a specific whitelisted Pod AND have a valid identity certificate

Q5: You enabled STRICT mTLS on a namespace and a monitoring service (not part of the mesh) can no longer scrape metrics from Pods. How do you fix this?

The problem: The monitoring service (e.g., Prometheus) is not part of the Istio mesh, so it doesn’t have an mTLS certificate. With STRICT mode, the meshed Pods reject plaintext connections from the monitor.

Three possible fixes:

Option 1: Add monitoring to the mesh (best)

# Inject Istio sidecar into monitoring namespace
kubectl label namespace monitoring istio-injection=enabled
kubectl rollout restart deployment prometheus -n monitoring

Now Prometheus has a sidecar and can make mTLS connections.

Option 2: Per-port PERMISSIVE for metrics port

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: allow-metrics-scraping
  namespace: production
spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    9090:
      mode: PERMISSIVE    # Metrics port accepts plaintext

This allows Prometheus to scrape port 9090 without mTLS while keeping all other ports STRICT.

Option 3: Exempt monitoring service account

This requires configuring Istio’s AuthorizationPolicy to allow the monitoring service, but for mTLS specifically, Option 1 or 2 is cleaner.

Recommendation: Option 1 is the most secure — add monitoring to the mesh so all connections are encrypted. Option 2 is acceptable when you can’t modify the monitoring setup.

---

Previous	Up	Next
Runtime Sandboxing	Pod Security	Pod Security Lab