KCSA — Service Mesh Security

6 minute read

Service Mesh Security

In a Kubernetes cluster, services communicate constantly — frontend calls backend, backend calls database, APIs call external services. By default, this traffic is unencrypted and unauthenticated within the cluster network. A service mesh solves this by providing automatic encryption, identity-based access control, and traffic observability without changing application code.

What Is a Service Mesh?

A service mesh is a dedicated infrastructure layer for managing service-to-service communication. It handles networking concerns — security, observability, reliability — transparently by intercepting all traffic between services.

Key characteristics:

  • Transparent — applications don’t need code changes to get mTLS, retries, or observability
  • Infrastructure layer — operates at the network level, separate from application logic
  • Sidecar pattern — typically uses a proxy sidecar injected into each pod
  • Policy-driven — defines communication rules as Kubernetes resources

A service mesh is NOT a replacement for NetworkPolicy — they operate at different layers and complement each other.

Service Mesh Architecture

A service mesh consists of two planes:

┌─────────────────────────────────────────────────────────┐
│                    Control Plane                         │
│   (istiod / linkerd-control-plane)                      │
│   - Certificate authority (issues mTLS certs)           │
│   - Configuration distribution                          │
│   - Policy management                                   │
│   - Telemetry collection                                │
└─────────────────────┬───────────────────────────────────┘
                      │ configures
          ┌───────────┼───────────┐
          ▼           ▼           ▼
     ┌─────────┐ ┌─────────┐ ┌─────────┐
     │ Pod A   │ │ Pod B   │ │ Pod C   │
     │┌───────┐│ │┌───────┐│ │┌───────┐│
     ││  App  ││ ││  App  ││ ││  App  ││
     │├───────┤│ │├───────┤│ │├───────┤│
     ││ Proxy ││ ││ Proxy ││ ││ Proxy ││
     ││(Envoy)││ ││(Envoy)││ ││(Envoy)││
     │└───────┘│ │└───────┘│ │└───────┘│
     └─────────┘ └─────────┘ └─────────┘
         Data Plane (sidecar proxies)

Two Planes

Plane Components Responsibilities
Control Plane istiod (Istio), linkerd-control-plane (Linkerd) Certificate management, configuration distribution, policy, telemetry
Data Plane Sidecar proxies (Envoy for Istio, linkerd2-proxy for Linkerd) Intercept traffic, enforce mTLS, apply policies, collect metrics

The sidecar proxy is automatically injected into every pod (via mutating admission webhook) and intercepts all inbound and outbound traffic.

mTLS — Mutual TLS

The primary security feature of a service mesh is mutual TLS (mTLS) — automatic encryption and mutual authentication between services.

How mTLS Works in a Mesh

  1. Identity issuance — the control plane issues a unique identity certificate (SPIFFE ID) to each workload
  2. Certificate distribution — certificates are pushed to sidecar proxies via secure channels
  3. Handshake — when Service A calls Service B, both sidecars perform a TLS handshake, each presenting its certificate
  4. Mutual verification — both sides verify the other’s identity against the certificate authority
  5. Encrypted communication — all subsequent traffic flows through an encrypted TLS tunnel
Pod A (Proxy) ←──── mTLS tunnel ────→ Pod B (Proxy)
     │                                      │
  presents                              presents
  cert for A                            cert for B
     │                                      │
  verifies B's cert                   verifies A's cert

mTLS Modes

Mode Behavior Use Case
STRICT Only mTLS connections allowed — plaintext rejected Production — full encryption enforced
PERMISSIVE Accept both mTLS and plaintext connections Migration — onboarding services gradually

STRICT mode should be the target for production environments. PERMISSIVE mode is used during migration when some services are not yet part of the mesh.

PeerAuthentication Resource (Istio)

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT   # Only accept mTLS connections in this namespace

PeerAuthentication can be applied at three scopes:

Scope Applied To
Mesh-wide root namespace (istio-system) — applies to all services
Namespace specific namespace — overrides mesh-wide for that namespace
Workload specific pod (via selector) — overrides namespace setting

Istio Security Features

Istio is the most feature-rich service mesh and the most commonly referenced in KCSA contexts:

Feature Resource Description
mTLS PeerAuthentication Automatic TLS between pods — STRICT or PERMISSIVE
Authorization AuthorizationPolicy Allow/deny rules for traffic based on source, destination, headers
JWT Validation RequestAuthentication Validate JSON Web Tokens on incoming requests
Peer Identity SPIFFE certificates Each workload gets a cryptographic identity

AuthorizationPolicy Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/production/sa/frontend"]
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/*"]

This policy allows the frontend service account to call GET and POST on /api/* paths of the backend service. All other traffic to backend is denied.

RequestAuthentication Example

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-validation
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-gateway
  jwtRules:
    - issuer: "https://auth.example.com"
      jwksUri: "https://auth.example.com/.well-known/jwks.json"

AuthorizationPolicy provides L7 (application layer) access control — you can make decisions based on HTTP methods, paths, headers, and source identity. This is far more granular than NetworkPolicy.

Linkerd Security Features

Linkerd takes a “secure by default” approach — simpler than Istio, with automatic mTLS enabled out of the box:

Feature Description
Automatic mTLS Enabled by default — no configuration needed
Identity Uses certificates issued by the Linkerd control plane (or external CA)
Authorization Server and ServerAuthorization resources for traffic control
Minimal footprint Written in Rust — lightweight proxy with low latency overhead
Zero-config security mTLS works immediately after mesh injection

Linkerd vs Istio at a Glance

Aspect Istio Linkerd
mTLS Configurable (STRICT/PERMISSIVE) On by default
Proxy Envoy (C++) linkerd2-proxy (Rust)
Complexity Feature-rich, more configuration Simpler, opinionated defaults
Authorization AuthorizationPolicy (very granular) Server/ServerAuthorization
JWT validation Built-in (RequestAuthentication) Requires external tooling
Resource overhead Higher Lower
CNCF Status Not CNCF (Google/IBM led) CNCF Graduated

For the KCSA exam, know that Linkerd is CNCF Graduated and focuses on simplicity and security by default, while Istio is more feature-rich but more complex.

Service Mesh vs NetworkPolicy

This is a key conceptual comparison for the exam:

Aspect NetworkPolicy Service Mesh (mTLS + AuthorizationPolicy)
OSI Layer L3/L4 (IP, port, protocol) L4/L7 (TLS, HTTP method, path, headers)
Encryption No — controls access, not confidentiality Yes — mTLS encrypts all traffic
Identity IP-based or label-based (pod selectors) Cryptographic identity (SPIFFE certificates)
Granularity Allow/deny by IP, port, namespace Allow/deny by service identity, HTTP method, path
Implementation CNI plugin (Calico, Cilium) Sidecar proxy (Envoy, linkerd2-proxy)
Overhead Minimal Additional proxy per pod (CPU, memory, latency)
Use without mesh Yes — built into Kubernetes Requires mesh installation
Mutual authentication No Yes — both sides verify identity

When You Need a Service Mesh

  • You need encryption between services (mTLS)
  • You need L7 policies (HTTP method/path-based access control)
  • You need mutual authentication (cryptographic service identity)
  • You need observability (distributed tracing, traffic metrics) without code changes

When NetworkPolicy Is Enough

  • You only need L3/L4 access control (block traffic by port/namespace)
  • Your threat model does not require in-cluster encryption
  • You want zero additional overhead per pod
  • Your cluster is small and traffic patterns are simple

In many production environments, both are used together — NetworkPolicy for coarse-grained L3/L4 segmentation, and a service mesh for fine-grained L7 policies and encryption.

Service Mesh Comparison

Feature Istio Linkerd Cilium Service Mesh
Proxy Envoy (sidecar) linkerd2-proxy (sidecar) eBPF (sidecarless) + Envoy
mTLS Yes (configurable) Yes (default on) Yes (via eBPF or Envoy)
L7 Policy AuthorizationPolicy Server/ServerAuthorization CiliumNetworkPolicy
Sidecar required Yes (ambient mode coming) Yes No (eBPF-native option)
CNCF Status Not CNCF Graduated Incubating (Cilium)
Complexity High Low Medium
Observability Kiali, Prometheus, Jaeger Linkerd Viz, Prometheus Hubble
Best for Feature-rich enterprise environments Simplicity, security by default eBPF-native, sidecar-free

Cilium Service Mesh is noteworthy because it can provide mesh capabilities without sidecars using eBPF — eliminating the per-pod proxy overhead. This is an emerging pattern in the ecosystem.

Key Takeaways for the Exam

  1. Service mesh = infrastructure layer for secure service-to-service communication
  2. mTLS provides encryption + mutual authentication — both sides verify identity via certificates
  3. STRICT mode rejects plaintext — use PERMISSIVE during migration
  4. Istio’s AuthorizationPolicy provides L7 access control (HTTP method, path, headers)
  5. PeerAuthentication controls mTLS mode (mesh-wide, namespace, or workload scope)
  6. Linkerd is CNCF Graduated, secure by default — mTLS on with no configuration
  7. Service mesh ≠ NetworkPolicy — mesh provides L7 + encryption, NetworkPolicy is L3/L4 only
  8. Sidecar proxies are injected via mutating admission webhook — transparent to the application
  9. Cilium Service Mesh is the sidecarless option using eBPF
  10. Use both together — NetworkPolicy for coarse-grained segmentation, mesh for fine-grained L7 + encryption

Next: Security Observability Stack

Updated:

You may also enjoy