CKS — Cluster Hardening Lab

11 minute read

Cluster Hardening — Hands-On Lab

This lab ties together all Cluster Hardening topics: RBAC auditing, ServiceAccount security, API server audit logging, and access restriction. Each exercise is designed to mirror CKS exam tasks.

Prerequisites

  • A kubeadm-based cluster (single control plane + at least one worker node)
  • SSH access to the control plane node
  • kubectl configured with cluster-admin access

Lab 1: Audit RBAC Permissions for the Default ServiceAccount

Objective: Discover what the default ServiceAccount in the default namespace can do.

Steps

# 1. Check permissions for the default ServiceAccount
kubectl auth can-i --list \
  --as=system:serviceaccount:default:default

# 2. Check if it can read Secrets
kubectl auth can-i get secrets \
  --as=system:serviceaccount:default:default

# 3. Check if it can create Pods
kubectl auth can-i create pods \
  --as=system:serviceaccount:default:default

# 4. Find all RoleBindings and ClusterRoleBindings that reference the default SA
kubectl get rolebindings -A -o json | jq -r '
  .items[] |
  select(.subjects[]? | .name == "default" and .kind == "ServiceAccount") |
  "\(.metadata.namespace)/\(.metadata.name) → \(.roleRef.name)"
'

kubectl get clusterrolebindings -o json | jq -r '
  .items[] |
  select(.subjects[]? | .name == "default" and .kind == "ServiceAccount") |
  "\(.metadata.name) → \(.roleRef.name)"
'

Expected Output

On a default cluster, the default ServiceAccount should have minimal permissions (only system-level access via system:discovery and similar RBAC rules). If you find any non-system bindings, they represent a security issue.

Lab 2: Create a Least-Privilege Role and RoleBinding

Objective: Create a ServiceAccount, Role, and RoleBinding that allows a workload to only read ConfigMaps in the app-ns namespace.

Steps

# 1. Create the namespace
kubectl create namespace app-ns

# 2. Create the ServiceAccount, Role, and RoleBinding
# Save as: lab2-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: config-reader
  namespace: app-ns
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: configmap-reader
  namespace: app-ns
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: config-reader-binding
  namespace: app-ns
subjects:
- kind: ServiceAccount
  name: config-reader
  namespace: app-ns
roleRef:
  kind: Role
  name: configmap-reader
  apiGroup: rbac.authorization.k8s.io

# 3. Apply the configuration
kubectl apply -f lab2-rbac.yaml

# 4. Verify: config-reader CAN read ConfigMaps
kubectl auth can-i get configmaps \
  --as=system:serviceaccount:app-ns:config-reader \
  -n app-ns
# Expected: yes

# 5. Verify: config-reader CANNOT read Secrets
kubectl auth can-i get secrets \
  --as=system:serviceaccount:app-ns:config-reader \
  -n app-ns
# Expected: no

# 6. Verify: config-reader CANNOT create Pods
kubectl auth can-i create pods \
  --as=system:serviceaccount:app-ns:config-reader \
  -n app-ns
# Expected: no

# 7. Verify: config-reader CANNOT access other namespaces
kubectl auth can-i get configmaps \
  --as=system:serviceaccount:app-ns:config-reader \
  -n default
# Expected: no

# 8. Full permission listing
kubectl auth can-i --list \
  --as=system:serviceaccount:app-ns:config-reader \
  -n app-ns

Expected Output

Resources                      Non-Resource URLs   Resource Names   Verbs
configmaps                     []                  []               [get list watch]
selfsubjectaccessreviews...    []                  []               [create]
selfsubjectrulesreviews...     []                  []               [create]
                               [/api/*]            []               [get]
                               [/healthz]          []               [get]
...

Lab 3: Disable ServiceAccount Token Auto-Mounting

Objective: Disable token auto-mount for the default ServiceAccount in app-ns and verify that Pods no longer have API access.

Steps

# 1. Patch the default ServiceAccount
kubectl patch serviceaccount default -n app-ns \
  -p '{"automountServiceAccountToken": false}'

# 2. Verify the patch
kubectl get serviceaccount default -n app-ns -o yaml
# Should show: automountServiceAccountToken: false

# 3. Create a test Pod using the default SA
kubectl run test-pod --image=alpine:3.19 -n app-ns \
  --command -- sleep 3600

# 4. Wait for the Pod to be running
kubectl wait --for=condition=Ready pod/test-pod -n app-ns --timeout=60s

# 5. Verify NO token is mounted
kubectl exec test-pod -n app-ns -- \
  ls /var/run/secrets/kubernetes.io/serviceaccount/ 2>&1
# Expected: ls: /var/run/secrets/kubernetes.io/serviceaccount/: No such file or directory

# 6. Verify the Pod cannot access the API
kubectl exec test-pod -n app-ns -- \
  wget -qO- --timeout=5 https://kubernetes.default.svc/api/v1/namespaces 2>&1
# Expected: connection refused or no credentials error

Now Create a Pod That DOES Need a Token

# lab3-token-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: api-aware-pod
  namespace: app-ns
spec:
  serviceAccountName: config-reader
  automountServiceAccountToken: true     # Override SA-level setting
  containers:
  - name: app
    image: alpine:3.19
    command: ["sleep", "3600"]

# Apply and verify token IS mounted (Pod-level override)
kubectl apply -f lab3-token-pod.yaml
kubectl wait --for=condition=Ready pod/api-aware-pod -n app-ns --timeout=60s
kubectl exec api-aware-pod -n app-ns -- \
  ls /var/run/secrets/kubernetes.io/serviceaccount/
# Expected: ca.crt  namespace  token

Lab 4: Create a Custom Audit Policy for Secrets Access

Objective: Write an audit policy that captures all Secrets operations at RequestResponse level while keeping general logging lightweight.

Steps

# 1. SSH to the control plane node
ssh controlplane

# 2. Create the audit policy directory
sudo mkdir -p /etc/kubernetes/audit

# 3. Create the audit policy
sudo tee /etc/kubernetes/audit/audit-policy.yaml > /dev/null <<'EOF'
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # Do not log health check endpoints
  - level: None
    nonResourceURLs:
    - "/healthz*"
    - "/readyz*"
    - "/livez*"

  # Do not log watch requests (very noisy)
  - level: None
    verbs: ["watch"]

  # Log all Secret operations with full request and response
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets"]

  # Log RBAC changes with full detail
  - level: RequestResponse
    resources:
    - group: "rbac.authorization.k8s.io"
      resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

  # Log Pod exec/attach at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/exec", "pods/attach"]

  # Catch-all: log everything else at Metadata level
  - level: Metadata
EOF

Verify the Policy Syntax

# Quick YAML syntax check
python3 -c "
import yaml, sys
with open('/etc/kubernetes/audit/audit-policy.yaml') as f:
    doc = yaml.safe_load(f)
print(f'Policy has {len(doc[\"rules\"])} rules')
print('Syntax OK')
"

Lab 5: Configure API Server with Audit Logging

Objective: Enable audit logging on the API server using the policy created in Lab 4.

Steps

# 1. Create the log directory
sudo mkdir -p /var/log/kubernetes/audit

# 2. Backup the current API server manifest
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml \
  /etc/kubernetes/kube-apiserver.yaml.backup

# 3. Edit the API server manifest
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add These Flags to the API Server Command

    # Add after existing flags in spec.containers[0].command:
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=7
    - --audit-log-maxbackup=3
    - --audit-log-maxsize=100

Add Volume Mounts

    # Add to spec.containers[0].volumeMounts:
    - name: audit-policy
      mountPath: /etc/kubernetes/audit
      readOnly: true
    - name: audit-logs
      mountPath: /var/log/kubernetes/audit

Add Volumes

  # Add to spec.volumes:
  - name: audit-policy
    hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
  - name: audit-logs
    hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate

Verify API Server Restart

# 3. Wait for the API server to restart (kubelet detects manifest change)
# This may take 30-60 seconds
sleep 30

# 4. Check if the API server is running
kubectl get pods -n kube-system -l component=kube-apiserver

# If kubectl doesn't respond, check with crictl:
sudo crictl ps | grep apiserver

# If the API server is crash-looping, check logs:
sudo crictl ps -a | grep apiserver
sudo crictl logs <container-id>

# 5. Verify audit logs are being written
sudo ls -la /var/log/kubernetes/audit/
sudo tail -3 /var/log/kubernetes/audit/audit.log | jq .

Generate Test Events

# Create a Secret
kubectl create secret generic lab-secret \
  --from-literal=password=s3cr3t -n app-ns

# Read the Secret
kubectl get secret lab-secret -n app-ns -o yaml

# Delete the Secret
kubectl delete secret lab-secret -n app-ns

# Check audit logs for these events
sudo cat /var/log/kubernetes/audit/audit.log | \
  jq 'select(.objectRef.resource == "secrets" and .objectRef.name == "lab-secret") |
  {verb, user: .user.username, response: .responseStatus.code}'

Expected Output

{"verb":"create","user":"kubernetes-admin","response":201}
{"verb":"get","user":"kubernetes-admin","response":200}
{"verb":"delete","user":"kubernetes-admin","response":200}

Lab 6: Verify RBAC Restrictions with kubectl auth can-i

Objective: Systematically verify RBAC permissions for different identities.

Steps

# 1. Create test ServiceAccounts with different privilege levels
kubectl create serviceaccount viewer-sa -n app-ns
kubectl create serviceaccount editor-sa -n app-ns
kubectl create serviceaccount admin-sa -n app-ns

# 2. Create Roles
cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: viewer-role
  namespace: app-ns
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: editor-role
  namespace: app-ns
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ns-admin-role
  namespace: app-ns
rules:
- apiGroups: ["", "apps", "batch"]
  resources: ["*"]
  verbs: ["*"]
EOF

# 3. Create RoleBindings
kubectl create rolebinding viewer-binding \
  --role=viewer-role \
  --serviceaccount=app-ns:viewer-sa \
  -n app-ns

kubectl create rolebinding editor-binding \
  --role=editor-role \
  --serviceaccount=app-ns:editor-sa \
  -n app-ns

kubectl create rolebinding admin-binding \
  --role=ns-admin-role \
  --serviceaccount=app-ns:admin-sa \
  -n app-ns

# 4. Run the verification matrix
echo "=== Viewer SA ==="
echo -n "get pods:       "; kubectl auth can-i get pods --as=system:serviceaccount:app-ns:viewer-sa -n app-ns
echo -n "create pods:    "; kubectl auth can-i create pods --as=system:serviceaccount:app-ns:viewer-sa -n app-ns
echo -n "get secrets:    "; kubectl auth can-i get secrets --as=system:serviceaccount:app-ns:viewer-sa -n app-ns
echo -n "delete deploy:  "; kubectl auth can-i delete deployments --as=system:serviceaccount:app-ns:viewer-sa -n app-ns

echo ""
echo "=== Editor SA ==="
echo -n "get pods:       "; kubectl auth can-i get pods --as=system:serviceaccount:app-ns:editor-sa -n app-ns
echo -n "create pods:    "; kubectl auth can-i create pods --as=system:serviceaccount:app-ns:editor-sa -n app-ns
echo -n "get secrets:    "; kubectl auth can-i get secrets --as=system:serviceaccount:app-ns:editor-sa -n app-ns
echo -n "delete deploy:  "; kubectl auth can-i delete deployments --as=system:serviceaccount:app-ns:editor-sa -n app-ns

echo ""
echo "=== Admin SA ==="
echo -n "get pods:       "; kubectl auth can-i get pods --as=system:serviceaccount:app-ns:admin-sa -n app-ns
echo -n "create pods:    "; kubectl auth can-i create pods --as=system:serviceaccount:app-ns:admin-sa -n app-ns
echo -n "get secrets:    "; kubectl auth can-i get secrets --as=system:serviceaccount:app-ns:admin-sa -n app-ns
echo -n "delete deploy:  "; kubectl auth can-i delete deployments --as=system:serviceaccount:app-ns:admin-sa -n app-ns

echo ""
echo "=== Cross-Namespace Check (should all be 'no') ==="
echo -n "admin-sa in default ns: "; kubectl auth can-i get pods --as=system:serviceaccount:app-ns:admin-sa -n default

Expected Output

=== Viewer SA ===
get pods:       yes
create pods:    no
get secrets:    no
delete deploy:  no

=== Editor SA ===
get pods:       yes
create pods:    yes
get secrets:    no
delete deploy:  yes

=== Admin SA ===
get pods:       yes
create pods:    yes
get secrets:    yes
delete deploy:  yes

=== Cross-Namespace Check (should all be 'no') ===
admin-sa in default ns: no

The admin-sa can access Secrets because the ns-admin-role uses resources: ["*"] — this is why wildcards should be used carefully.

Cleanup

# Remove lab resources
kubectl delete namespace app-ns

# Remove audit configuration (if desired)
sudo rm -rf /etc/kubernetes/audit
sudo rm -rf /var/log/kubernetes/audit

# Restore original API server manifest (if you backed it up)
# sudo cp /etc/kubernetes/kube-apiserver.yaml.backup \
#   /etc/kubernetes/manifests/kube-apiserver.yaml

Note: If you remove the audit configuration, also remove the --audit-* flags and corresponding volume mounts from the API server manifest. Otherwise the API server will fail to start.

Practice Questions

Q1. You need to verify that the backend-sa ServiceAccount in the production namespace cannot read Secrets. Write the command.

Show Answer 
kubectl auth can-i get secrets \
  --as=system:serviceaccount:production:backend-sa \
  -n production
# Expected output: no

To also check list and watch:

kubectl auth can-i list secrets \
  --as=system:serviceaccount:production:backend-sa \
  -n production

kubectl auth can-i watch secrets \
  --as=system:serviceaccount:production:backend-sa \
  -n production

Q2. Write an audit policy that logs Secret operations at RequestResponse level and everything else at Metadata level. Include the API server flags needed to enable it.

Show Answer

Audit Policy (/etc/kubernetes/audit/audit-policy.yaml):

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: None
    nonResourceURLs:
    - "/healthz*"
    - "/readyz*"
    - "/livez*"
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets"]
  - level: Metadata

API Server Flags:

- --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
- --audit-log-path=/var/log/kubernetes/audit/audit.log
- --audit-log-maxage=30
- --audit-log-maxbackup=3
- --audit-log-maxsize=100

Plus the corresponding volume mounts for /etc/kubernetes/audit (readOnly) and /var/log/kubernetes/audit.

Q3. After patching the default ServiceAccount with automountServiceAccountToken: false, a new Pod is created with automountServiceAccountToken: true in its spec. Will the token be mounted?

Show Answer

Yes. The Pod-level setting takes precedence over the ServiceAccount-level setting when explicitly set. The precedence is:

  • Pod spec automountServiceAccountToken: true → token IS mounted (regardless of SA setting)
  • Pod spec automountServiceAccountToken: false → token NOT mounted (regardless of SA setting)
  • Pod spec not set → uses SA setting
  • Neither set → defaults to true (mounted)

This is why defense in depth is important: use SA-level defaults AND enforce via admission policy.

Q4. You find a ClusterRoleBinding that grants edit ClusterRole to system:serviceaccounts group. What is the impact?

Show Answer

The system:serviceaccounts group includes every ServiceAccount in every namespace. The built-in edit ClusterRole grants create, update, delete on most namespace-scoped resources.

Impact: Every Pod in the cluster (since every Pod has a ServiceAccount) can modify resources in its namespace — create Pods, modify ConfigMaps, etc. A compromised container could:

  • Deploy malicious workloads
  • Modify running Deployments
  • Access ConfigMaps with sensitive data

Remediation:

# Find the binding
kubectl get clusterrolebindings -o json | jq -r '
  .items[] | select(.subjects[]?.name == "system:serviceaccounts") |
  .metadata.name
'

# Delete it
kubectl delete clusterrolebinding <name>

Q5. The API server is not starting after you added audit logging. The crictl logs show: Error: loading audit policy file: open /etc/kubernetes/audit/policy.yaml: no such file or directory. What went wrong?

Show Answer

The error message shows the API server is looking for /etc/kubernetes/audit/policy.yaml but the file doesn’t exist at that path inside the container. Common causes:

  1. File name mismatch: The --audit-policy-file flag points to policy.yaml but the actual file is named audit-policy.yaml (or vice versa).

  2. Volume mount missing: The hostPath volume for /etc/kubernetes/audit may not be mounted into the container.

  3. File doesn’t exist on host: The file wasn’t created at the correct path on the host filesystem.

Fix:

# 1. Verify the file exists on the host
ls -la /etc/kubernetes/audit/

# 2. Ensure the --audit-policy-file flag matches the actual filename
grep audit-policy-file /etc/kubernetes/manifests/kube-apiserver.yaml

# 3. Ensure the volume mount exists in the manifest
grep -A3 'audit' /etc/kubernetes/manifests/kube-apiserver.yaml

# 4. Correct the flag or rename the file to match
Previous Up Next
API Server Audit CKS Home Minimize Microservice Vulnerabilities

Updated:

You may also enjoy