CKS — Immutable Containers

9 minute read

Immutable Containers

An immutable container cannot be modified at runtime. No files can be written, no packages installed, no binaries replaced. If an attacker gains access to an immutable container, they cannot persist malware, install tools, or tamper with the application. Immutability is a core CKS exam topic — expect to configure a Pod with full immutability settings.

Why Immutability Matters

Without Immutability With Immutability
Attacker writes cryptominer to /tmp Write denied — read-only filesystem
Attacker installs nmap via apt-get Package manager cannot write to disk
Attacker modifies /etc/passwd /etc is read-only
Attacker creates reverse shell script Cannot create new files anywhere
Attacker modifies application code Application directory is read-only

Immutability implements the principle of least privilege at the filesystem level. If your container doesn’t need to write files, don’t let it.

The Three Pillars of Container Immutability

  1. Read-only root filesystem — prevents all writes to the container filesystem
  2. Dropped capabilities — removes kernel-level privileges
  3. Non-root execution — runs as an unprivileged user

All three must be combined for true immutability.

readOnlyRootFilesystem

The most direct immutability control. When set to true, the entire container root filesystem becomes read-only.

apiVersion: v1
kind: Pod
metadata:
  name: readonly-pod
  namespace: default
spec:
  containers:
  - name: app
    image: nginx:1.25
    securityContext:
      readOnlyRootFilesystem: true

What happens:

# Inside the container:
$ touch /tmp/test
touch: cannot touch '/tmp/test': Read-only file system

$ apt-get update
E: Could not open lock file - open (30: Read-only file system)

Problem: Many applications need to write to certain directories — /tmp, /var/run, /var/cache. The container will crash if it cannot write to required paths.

Using emptyDir for Writable Temporary Directories

The solution: mount emptyDir volumes on paths that need to be writable. An emptyDir volume is a temporary directory that exists only for the Pod’s lifetime.

apiVersion: v1
kind: Pod
metadata:
  name: immutable-nginx
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /var/cache/nginx
    - name: run
      mountPath: /var/run
  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}
  - name: run
    emptyDir: {}

Now:

  • The root filesystem is read-only
  • /tmp, /var/cache/nginx, and /var/run are writable (ephemeral storage)
  • An attacker cannot write to /etc, /usr, /bin, or any other path

Dropping All Capabilities

Linux capabilities grant specific privileges to processes. By default, containers get a set of capabilities. Drop them all and add back only what is needed:

securityContext:
  capabilities:
    drop:
      - ALL

What this prevents:

Capability What It Allows Dropped?
NET_RAW Raw sockets (packet sniffing) Yes — dropped
SYS_ADMIN Mount filesystems, modify namespaces Yes — dropped
SYS_PTRACE Trace processes (injection attacks) Yes — dropped
NET_BIND_SERVICE Bind to ports < 1024 Yes — dropped
CHOWN Change file ownership Yes — dropped
SETUID / SETGID Change user/group ID Yes — dropped
DAC_OVERRIDE Bypass file permission checks Yes — dropped

If your application needs a specific capability, add it back:

securityContext:
  capabilities:
    drop:
      - ALL
    add:
      - NET_BIND_SERVICE    # Only if needed

Preventing Privilege Escalation

The allowPrivilegeEscalation field prevents a process from gaining more privileges than its parent:

securityContext:
  allowPrivilegeEscalation: false

This blocks:

  • setuid binaries (like sudo or su)
  • Kernel exploits that escalate to root
  • Any mechanism that would increase the process’s privilege level

Running as Non-Root

Force the container to run as a non-root user:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000

What this prevents:

  • Root-level file access
  • Root-level kernel interaction
  • Many container escape techniques that require root

Seccomp Profile — RuntimeDefault

Seccomp (Secure Computing Mode) restricts which syscalls a container can make. The RuntimeDefault profile blocks dangerous syscalls while allowing normal operation:

securityContext:
  seccompProfile:
    type: RuntimeDefault

Blocked by RuntimeDefault:

  • mount — no filesystem mounting
  • reboot — cannot reboot the host
  • ptrace — no process tracing
  • kexec_load — cannot load a new kernel
  • bpf — cannot load eBPF programs

The Fully Immutable Pod — Complete YAML

This is the gold standard for CKS exam questions. Memorize this pattern:

apiVersion: v1
kind: Pod
metadata:
  name: fully-immutable-pod
  namespace: default
  labels:
    app: secure-app
spec:
  automountServiceAccountToken: false    # Don't mount SA token
  containers:
  - name: app
    image: mycompany/app:v2.1
    ports:
    - containerPort: 8080
    securityContext:
      readOnlyRootFilesystem: true       # Read-only filesystem
      allowPrivilegeEscalation: false    # No privilege escalation
      runAsNonRoot: true                 # Must run as non-root
      runAsUser: 1000                    # Specific non-root UID
      runAsGroup: 1000                   # Specific non-root GID
      capabilities:
        drop:
          - ALL                          # Drop all capabilities
      seccompProfile:
        type: RuntimeDefault             # Restrict syscalls
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /var/cache
    resources:
      requests:
        memory: "64Mi"
        cpu: "100m"
      limits:
        memory: "128Mi"
        cpu: "200m"
  volumes:
  - name: tmp
    emptyDir:
      sizeLimit: 100Mi
  - name: cache
    emptyDir:
      sizeLimit: 50Mi

Security settings breakdown:

Setting Value Effect
readOnlyRootFilesystem true No writes to container filesystem
allowPrivilegeEscalation false No setuid, no privilege gain
runAsNonRoot true Pod rejected if image runs as root
runAsUser 1000 Runs as UID 1000
capabilities.drop ALL No Linux capabilities
seccompProfile RuntimeDefault Restricted syscalls
automountServiceAccountToken false No API server access
emptyDir.sizeLimit Set Bounded temp storage

Preventing exec Into Containers

kubectl exec is one of the most common attack vectors. While you cannot directly disable it at the Pod level, you can make it useless:

Approach 1: readOnlyRootFilesystem — even if an attacker execs in, they cannot write anything.

Approach 2: Drop all capabilities + non-root — limits what commands can do:

# Inside a locked-down container:
$ whoami
appuser
$ id
uid=1000(appuser) gid=1000(appgroup)
$ cat /etc/shadow
cat: /etc/shadow: Permission denied
$ apt-get update
bash: apt-get: command not found    # Distroless image

Approach 3: Use distroless images — no shell, no package manager, no utilities:

# Distroless image — no shell exists
image: gcr.io/distroless/static:nonroot

If there is no shell binary in the image, kubectl exec cannot spawn one:

$ kubectl exec -it pod -- /bin/sh
OCI runtime exec failed: exec failed: unable to start container process:
exec: "/bin/sh": stat /bin/sh: no such file or directory

Combining with Pod Security Admission (PSA)

The Restricted PSA profile enforces most immutability settings at the namespace level:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest

What PSA Restricted requires:

Requirement Status
runAsNonRoot: true Enforced
allowPrivilegeEscalation: false Enforced
capabilities.drop: ALL Enforced
seccompProfile: RuntimeDefault or Localhost Enforced
No hostNetwork, hostPID, hostIPC Enforced
No privileged containers Enforced
readOnlyRootFilesystem NOT enforced — must set manually

Important: PSA Restricted does NOT enforce readOnlyRootFilesystem. You must always set this in your Pod spec explicitly.

Deployment with Immutable Containers

For production workloads, apply immutability at the Deployment level:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: secure-app
  template:
    metadata:
      labels:
        app: secure-app
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: app
        image: mycompany/app:v2.1
        ports:
        - containerPort: 8080
        securityContext:
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - ALL
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"
      volumes:
      - name: tmp
        emptyDir: {}

Testing Immutability

After deploying an immutable container, verify the controls work:

# Deploy the pod
kubectl apply -f immutable-pod.yaml

# Verify the pod is running
kubectl get pod fully-immutable-pod

# Try to write to the root filesystem — should fail
kubectl exec fully-immutable-pod -- touch /testfile
# Error: touch: cannot touch '/testfile': Read-only file system

# Try to write to /tmp — should succeed (emptyDir)
kubectl exec fully-immutable-pod -- touch /tmp/testfile
# (no error)

# Verify running as non-root
kubectl exec fully-immutable-pod -- id
# uid=1000 gid=1000

# Try to install a package — should fail
kubectl exec fully-immutable-pod -- apt-get update
# Read-only file system (or command not found for distroless)

Common Application Writable Paths

Different applications need different writable paths. Here are common ones:

Application Writable Paths Needed
nginx /var/cache/nginx, /var/run, /tmp
Node.js /tmp
Python/Flask /tmp
Java/Spring /tmp
Redis /data
PostgreSQL /var/lib/postgresql/data, /var/run/postgresql

Mount emptyDir volumes for each required writable path.

Security Impact Summary

Attack Vector Without Immutability With Full Immutability
Write malware to disk Possible Blocked — read-only FS
Install attack tools Possible Blocked — read-only FS
Modify config files Possible Blocked — read-only FS
Escalate to root Possible Blocked — no privilege escalation
Use setuid binaries Possible Blocked — capabilities dropped
Mount host filesystem Possible Blocked — seccomp + no SYS_ADMIN
Access API server Possible Blocked — no SA token mounted
Run as root Possible Blocked — runAsNonRoot

Practice Questions

Question 1

Write a Pod spec for an nginx container that is fully immutable. Ensure nginx can still write to its required directories.

Show Answer 
apiVersion: v1
kind: Pod
metadata:
  name: immutable-nginx
spec:
  automountServiceAccountToken: false
  containers:
  - name: nginx
    image: nginx:1.25
    ports:
    - containerPort: 80
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      runAsNonRoot: false              # nginx master starts as root
      capabilities:
        drop:
          - ALL
        add:
          - NET_BIND_SERVICE           # Needed for port 80
      seccompProfile:
        type: RuntimeDefault
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /var/cache/nginx
    - name: run
      mountPath: /var/run
  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}
  - name: run
    emptyDir: {}

Note: The default nginx image runs as root initially (to bind port 80). For full non-root, use nginxinc/nginx-unprivileged which listens on port 8080.

Question 2

A Pod with readOnlyRootFilesystem: true is crashing because the application writes to /var/log/app. How do you fix it while maintaining immutability?

Show Answer

Add an emptyDir volume mounted at /var/log/app:

spec:
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: app-logs
      mountPath: /var/log/app
  volumes:
  - name: app-logs
    emptyDir:
      sizeLimit: 100Mi

The root filesystem remains read-only, but /var/log/app uses ephemeral storage from the emptyDir volume.

Question 3

Does the PSA Restricted profile enforce readOnlyRootFilesystem?

Show Answer

No. The Pod Security Admission (PSA) Restricted profile does not enforce readOnlyRootFilesystem. It enforces:

  • runAsNonRoot: true
  • allowPrivilegeEscalation: false
  • capabilities.drop: ALL
  • seccompProfile: RuntimeDefault or Localhost
  • No host namespaces or privileged containers

You must set readOnlyRootFilesystem: true explicitly in your Pod spec. This is a common exam gotcha — students assume PSA Restricted covers everything.

Question 4

What is the benefit of using a distroless container image for security?

Show Answer

Distroless images contain only the application and its runtime dependencies — no shell, no package manager, no OS utilities.

Security benefits:

  • kubectl exec cannot spawn a shell (no /bin/sh or /bin/bash)
  • Attackers cannot install tools (apt-get, curl, wget don’t exist)
  • Smaller attack surface — fewer binaries means fewer potential vulnerabilities
  • Smaller image size — faster pulls, less storage

Example:

image: gcr.io/distroless/static:nonroot    # No shell at all
image: gcr.io/distroless/java17:nonroot    # Java runtime only
image: gcr.io/distroless/python3:nonroot   # Python runtime only

Combined with readOnlyRootFilesystem, a distroless image makes post-exploitation extremely difficult.

Question 5

Write the securityContext block that implements the maximum practical immutability settings for a CKS exam scenario.

Show Answer

Pod-level securityContext:

spec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault

Container-level securityContext:

  containers:
  - name: app
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

Combined effect:

  • No root execution
  • No filesystem writes (except emptyDir mounts)
  • No privilege escalation
  • No Linux capabilities
  • Restricted syscalls via seccomp
  • No service account token mounted

Next: Threat Detection →

Updated:

You may also enjoy