CKS — Runtime Security Lab

Runtime Security Lab

This hands-on lab walks you through Falco installation, custom rule creation, immutable container deployment, and real-time threat detection. Complete all exercises to build the muscle memory needed for CKS exam questions on runtime security.

Prerequisites:

• A Kubernetes cluster (single-node or multi-node)
• SSH access to a cluster node (for Falco installation)
• kubectl configured and working

---

Lab 1 — Install Falco on a Node

Objective: Install Falco from a DEB package, start the service, and verify it is running.

Step 1 — SSH into the node:

ssh node01

Step 2 — Add the Falco repository:

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
  sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/falcosecurity.list

Step 3 — Install Falco:

sudo apt-get update
sudo apt-get install -y falco

When prompted for the driver, select kmod (kernel module).

Step 4 — Start and enable Falco:

sudo systemctl start falco
sudo systemctl enable falco

Step 5 — Verify Falco is running:

sudo systemctl status falco

Expected output:

● falco.service - Falco: Container Native Runtime Security
     Loaded: loaded (/usr/lib/systemd/system/falco.service; enabled)
     Active: active (running) since ...

Step 6 — Check Falco version:

falco --version

---

Lab 2 — Verify Falco Is Detecting Events

Objective: Trigger a Falco alert by exec-ing into a container, then find the alert in logs.

Step 1 — Create a test pod:

kubectl run nginx-test --image=nginx:1.25
kubectl wait --for=condition=Ready pod/nginx-test --timeout=60s

Step 2 — Exec into the pod (triggers “Terminal shell in container”):

kubectl exec -it nginx-test -- /bin/bash

Step 3 — Inside the container, run a few commands:

whoami
cat /etc/shadow
ls /
exit

Step 4 — Check Falco logs on the node:

# SSH to the node if not already there
journalctl -u falco --since "5 minutes ago" | grep -i "shell"

Expected output:

Notice A shell was spawned in a container with an attached terminal
  (user=root user_loginuid=-1 container_id=abc123
   container_name=nginx-test image=nginx:1.25
   shell=bash parent=runc cmdline=bash terminal=34816
   k8s.ns.name=default k8s.pod.name=nginx-test)

Step 5 — Check for sensitive file read alert:

journalctl -u falco --since "5 minutes ago" | grep -i "sensitive"

Expected output:

Warning Sensitive file opened for reading by non-trusted program
  (user=root command=cat /etc/shadow file=/etc/shadow
   container=nginx-test image=nginx:1.25)

Verification: You should see at least two alerts — one for the shell spawn, one for the sensitive file read.

---

Lab 3 — Create a Custom Falco Rule for Namespace-Specific Shell Detection

Objective: Write a custom rule that detects shell spawning specifically in the production namespace with CRITICAL priority.

Step 1 — Create the production namespace:

kubectl create namespace production

Step 2 — Edit the custom rules file on the node:

sudo tee -a /etc/falco/falco_rules.local.yaml > /dev/null <<'EOF'

- rule: Shell in production namespace
  desc: Detect shell spawned in a container within the production namespace
  condition: >
    spawned_process and container
    and proc.name in (bash, sh, zsh, dash, ash)
    and proc.tty != 0
    and k8s.ns.name = "production"
  output: >
    PRODUCTION ALERT - Shell spawned
    (user=%user.name shell=%proc.name command=%proc.cmdline
     container=%container.name image=%container.image.repository
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: CRITICAL
  tags: [container, shell, production]
EOF

Step 3 — Validate and restart Falco:

sudo falco --validate /etc/falco/falco_rules.local.yaml
sudo systemctl restart falco
sudo systemctl status falco

Step 4 — Test the rule:

# Create a pod in the production namespace
kubectl run prod-nginx --image=nginx:1.25 -n production
kubectl wait --for=condition=Ready pod/prod-nginx -n production --timeout=60s

# Exec into it
kubectl exec -it prod-nginx -n production -- /bin/bash
exit

Step 5 — Verify the custom alert:

journalctl -u falco --since "2 minutes ago" | grep "PRODUCTION ALERT"

Expected output:

Critical PRODUCTION ALERT - Shell spawned
  (user=root shell=bash command=bash
   container=prod-nginx image=nginx:1.25
   namespace=production pod=prod-nginx)

Notice the priority is Critical (not just Notice/Warning like the default rule).

---

Lab 4 — Create a Custom Rule to Detect Sensitive File Access

Objective: Write a rule that detects when any process reads /etc/shadow or /etc/passwd inside a container.

Step 1 — Add the rule:

sudo tee -a /etc/falco/falco_rules.local.yaml > /dev/null <<'EOF'

- rule: Sensitive credential file access
  desc: Detect read access to credential files in containers
  condition: >
    open_read and container
    and fd.name in (/etc/shadow, /etc/passwd, /etc/gshadow)
    and not proc.name in (sshd, login, passwd)
  output: >
    CREDENTIAL FILE READ (user=%user.name command=%proc.cmdline
     file=%fd.name container=%container.name image=%container.image.repository
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: ERROR
  tags: [container, filesystem, credential_access]
EOF

Step 2 — Restart Falco:

sudo systemctl restart falco

Step 3 — Trigger the rule:

kubectl exec nginx-test -- cat /etc/shadow
kubectl exec nginx-test -- cat /etc/passwd

Step 4 — Verify:

journalctl -u falco --since "2 minutes ago" | grep "CREDENTIAL FILE"

Expected output:

Error CREDENTIAL FILE READ (user=root command=cat /etc/shadow
  file=/etc/shadow container=nginx-test image=nginx:1.25
  namespace=default pod=nginx-test)

---

Lab 5 — Deploy an Immutable Container

Objective: Create a fully immutable Pod with readOnlyRootFilesystem, dropped capabilities, non-root user, and emptyDir for writable temp paths.

Step 1 — Create the Pod manifest:

cat <<'EOF' > /tmp/immutable-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: immutable-app
  namespace: default
  labels:
    app: immutable-app
spec:
  automountServiceAccountToken: false
  containers:
  - name: app
    image: nginx:1.25
    ports:
    - containerPort: 80
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
        add:
          - NET_BIND_SERVICE
      seccompProfile:
        type: RuntimeDefault
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /var/cache/nginx
    - name: run
      mountPath: /var/run
  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}
  - name: run
    emptyDir: {}
EOF

Step 2 — Deploy the Pod:

kubectl apply -f /tmp/immutable-pod.yaml
kubectl wait --for=condition=Ready pod/immutable-app --timeout=60s

Step 3 — Verify the Pod is running:

kubectl get pod immutable-app -o wide

---

Lab 6 — Test Immutability

Objective: Verify the immutable container rejects filesystem writes and confirm writable emptyDir paths work.

Test 1 — Write to root filesystem (should FAIL):

kubectl exec immutable-app -- touch /testfile

Expected output:

touch: cannot touch '/testfile': Read-only file system
command terminated with exit code 1

Test 2 — Write to /etc (should FAIL):

kubectl exec immutable-app -- touch /etc/newfile

Expected output:

touch: cannot touch '/etc/newfile': Read-only file system
command terminated with exit code 1

Test 3 — Install a package (should FAIL):

kubectl exec immutable-app -- apt-get update

Expected output:

Reading package lists...
E: Could not open lock file /var/lib/apt/lists/lock - open (30: Read-only file system)

Test 4 — Write to /tmp (should SUCCEED — emptyDir):

kubectl exec immutable-app -- touch /tmp/testfile
kubectl exec immutable-app -- ls /tmp/testfile

Expected output:

/tmp/testfile

Test 5 — Write to /var/cache/nginx (should SUCCEED — emptyDir):

kubectl exec immutable-app -- touch /var/cache/nginx/testfile
kubectl exec immutable-app -- ls /var/cache/nginx/testfile

Expected output:

/var/cache/nginx/testfile

Test 6 — Verify capabilities are dropped:

# This requires capsh — may not be available in all images
kubectl exec immutable-app -- cat /proc/1/status | grep -i cap

Test 7 — Verify no service account token is mounted:

kubectl exec immutable-app -- ls /var/run/secrets/kubernetes.io/serviceaccount/

Expected output:

ls: cannot access '/var/run/secrets/kubernetes.io/serviceaccount/': No such file or directory

---

Lab 7 — Test Falco Detection with Immutable Container

Objective: Exec into the immutable container, attempt to read /etc/shadow, and verify Falco captures the alert — even though the container is immutable.

Step 1 — Exec into the immutable container:

kubectl exec -it immutable-app -- /bin/bash

Step 2 — Inside the container, try to read /etc/shadow:

cat /etc/shadow

Note: Even with readOnlyRootFilesystem, the existing files on the root filesystem (like /etc/shadow) can still be read. readOnlyRootFilesystem only prevents writes. The read still triggers Falco’s sensitive file detection.

Step 3 — Try to write a file (will fail because of read-only filesystem):

touch /malware.sh
# Read-only file system

Step 4 — Exit the container:

exit

Step 5 — Check Falco alerts on the node:

# Shell alert
journalctl -u falco --since "5 minutes ago" | grep "immutable-app" | grep -i "shell"

# Sensitive file read alert  
journalctl -u falco --since "5 minutes ago" | grep "immutable-app" | grep -i "sensitive\|shadow\|CREDENTIAL"

Expected alerts:

Critical PRODUCTION ALERT - Shell spawned (...container=immutable-app...)
Error CREDENTIAL FILE READ (...file=/etc/shadow container=immutable-app...)

Key takeaway: Immutability prevents writes but does not prevent reads or process execution. Falco complements immutability by detecting the suspicious read and exec activity. Together, they provide defense in depth.

---

Cleanup

# Delete test pods
kubectl delete pod nginx-test --ignore-not-found
kubectl delete pod prod-nginx -n production --ignore-not-found
kubectl delete pod immutable-app --ignore-not-found

# Delete production namespace
kubectl delete namespace production --ignore-not-found

# Remove custom rules (optional — keep for reference)
# sudo truncate -s 0 /etc/falco/falco_rules.local.yaml
# sudo systemctl restart falco

---

Lab Summary

Lab	Skill Tested	CKS Relevance
1	Install Falco from DEB package	Direct exam task
2	Verify Falco detects events	Understand alert format and log location
3	Write namespace-specific Falco rule	Custom rule creation — likely exam question
4	Write sensitive file detection rule	Custom rule creation
5	Deploy immutable container	Core exam skill
6	Test immutability controls	Verify readOnlyRootFilesystem + emptyDir
7	Combine Falco + immutability	Defense in depth — detection + prevention

---

Practice Questions

Question 1

After installing Falco, what single command verifies that the Falco service is running?

Show Answer

sudo systemctl status falco

Look for Active: active (running) in the output. Alternatively:

systemctl is-active falco

This returns active if running.

Question 2

You created a custom Falco rule but Falco won’t start after restarting. What is the first diagnostic step?

Show Answer

Check the journal logs for parsing errors:

journalctl -u falco --since "2 minutes ago"

Or validate the rule file directly:

sudo falco --validate /etc/falco/falco_rules.local.yaml

The most common cause is YAML syntax errors — incorrect indentation, missing quotes, or invalid field names.

Question 3

A Pod has readOnlyRootFilesystem: true but needs to write to /var/log/app. How do you enable this while keeping the root filesystem read-only?

Show Answer

Add an emptyDir volume mounted at /var/log/app:

spec:
  containers:
  - name: app
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: app-logs
      mountPath: /var/log/app
  volumes:
  - name: app-logs
    emptyDir: {}

The emptyDir provides writable ephemeral storage at that specific path. All other paths remain read-only.

Question 4

Can readOnlyRootFilesystem prevent an attacker from reading /etc/shadow?

Show Answer

No. readOnlyRootFilesystem only prevents writes to the container filesystem. Existing files like /etc/shadow can still be read.

To prevent reading /etc/shadow:

• Use a distroless image (no /etc/shadow present)
• Run as non-root (runAsNonRoot: true) so the process lacks read permission
• Use Falco to detect the read and alert on it

This is why immutability and runtime detection (Falco) complement each other — immutability prevents writes, Falco detects reads and execution.

Question 5

Write the complete securityContext block for a container that is fully locked down for CKS exam purposes.

Show Answer

Pod-level:

spec:
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault

Container-level:

  containers:
  - name: app
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

Combined effect:

• Read-only filesystem (no writes)
• No privilege escalation
• No Linux capabilities
• Non-root execution
• Seccomp restricts dangerous syscalls
• No service account token mounted

Add emptyDir volumes for any paths the application needs to write to.

---

← Back to Runtime Security & Falco