CKS — Audit Logging Lab

11 minute read

Audit Logging Lab

CKS Exam Domain: Domain 6 — Monitoring, Logging & Runtime Security (20%)

This lab walks you through the complete audit logging workflow — from writing an audit policy to configuring the API server, generating events, and analyzing the results with jq. These exercises mirror the kind of multi-step tasks you will encounter on the CKS exam.

Prerequisites

  • A kubeadm-based Kubernetes cluster (single or multi-node)
  • SSH access to the control plane node with root/sudo privileges
  • jq installed on the control plane node
  • Familiarity with editing kube-apiserver static Pod manifests

Exercise 1: Create an Audit Policy File

Objective: Write an audit policy that logs secrets at Request level, pods at Metadata level, and includes a Metadata catch-all.

Steps

  1. Create the audit directory:
sudo mkdir -p /etc/kubernetes/audit
  1. Create the audit policy file:
cat <<'EOF' | sudo tee /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
omitManagedFields: true
rules:
  # Don't log health check endpoints
  - level: None
    nonResourceURLs:
      - "/healthz*"
      - "/readyz*"
      - "/livez*"

  # Don't log events resource (noisy)
  - level: None
    resources:
      - group: ""
        resources: ["events"]

  # Log Secret access at Request level (captures secret values)
  - level: Request
    resources:
      - group: ""
        resources: ["secrets"]

  # Log Pod operations at Metadata level
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]

  # Catch-all: log everything else at Metadata
  - level: Metadata
EOF
  1. Validate the YAML syntax:
python3 -c "import yaml; yaml.safe_load(open('/etc/kubernetes/audit/audit-policy.yaml'))" && echo "YAML is valid"

Expected Output

YAML is valid

The file should exist at /etc/kubernetes/audit/audit-policy.yaml with correct YAML syntax and the rules in the order specified.

Exercise 2: Configure the API Server with Audit Logging

Objective: Modify the kube-apiserver manifest to enable audit logging with the policy from Exercise 1.

Steps

  1. Back up the current API server manifest:
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak
  1. Create the audit log directory:
sudo mkdir -p /var/log/kubernetes/audit
  1. Edit the API server manifest:
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
  1. Add the audit flags to the kube-apiserver command section (add these lines after the existing flags):
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
  1. Add the volumeMounts (inside the containers[0].volumeMounts section):
    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log
  1. Add the volumes (inside the spec.volumes section):
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log

  1. Save and exit. The kubelet will detect the change and restart the API server.

  2. Wait for the API server to restart:

# Wait until the API server responds
echo "Waiting for API server..."
until kubectl get nodes &>/dev/null; do
  sleep 2
  echo -n "."
done
echo " Ready!"
  1. Verify the API server has audit flags:
ps aux | grep kube-apiserver | grep audit-policy-file

Expected Output

root  ... kube-apiserver --advertise-address=... --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml --audit-log-path=/var/log/kubernetes/audit/audit.log ...

Troubleshooting

If the API server does not come back:

# Check kubelet logs for the error
sudo journalctl -u kubelet --since "3 minutes ago" | grep -i "error\|fail" | tail -20

# Common errors:
# - YAML syntax error in kube-apiserver.yaml → restore from backup
# - Policy file not found → verify the volume mount path
# - Permission denied → verify directory ownership

# Restore from backup if needed
sudo cp /tmp/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml

Exercise 3: Generate Audit Events

Objective: Perform various API operations to generate audit events, then verify they were logged.

Steps

  1. Create a secret:
kubectl create secret generic audit-test-secret \
  --from-literal=password=SuperSecret123 \
  -n default
  1. Read the secret:
kubectl get secret audit-test-secret -n default -o yaml
  1. Create a pod and exec into it:
kubectl run audit-test-pod --image=nginx:1.25 --restart=Never -n default

# Wait for the pod to be running
kubectl wait --for=condition=Ready pod/audit-test-pod -n default --timeout=60s

# Exec into the pod
kubectl exec -it audit-test-pod -n default -- whoami
  1. Create a role and rolebinding:
kubectl create role audit-test-role \
  --verb=get,list \
  --resource=pods \
  -n default

kubectl create rolebinding audit-test-binding \
  --role=audit-test-role \
  --serviceaccount=default:default \
  -n default
  1. Delete a resource:
kubectl delete secret audit-test-secret -n default
  1. Verify audit events are being written:
# Check log file exists and has content
ls -la /var/log/kubernetes/audit/audit.log
wc -l /var/log/kubernetes/audit/audit.log

Expected Output

The audit log file should exist and contain multiple lines (JSON events). Each line is one audit event.

Exercise 4: Analyze Audit Logs with jq

Objective: Use jq to find specific events in the audit log generated from Exercise 3.

Steps

  1. Find all Secret access events:
cat /var/log/kubernetes/audit/audit.log | jq 'select(.objectRef.resource=="secrets") | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  verb: .verb,
  name: .objectRef.name,
  namespace: .objectRef.namespace,
  code: .responseStatus.code
}'

Expected: You should see events for create, get, and delete operations on audit-test-secret.

  1. Find the secret creation event specifically:
cat /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  .objectRef.name=="audit-test-secret" and
  .verb=="create"
) | {
  user: .user.username,
  verb: .verb,
  name: .objectRef.name,
  code: .responseStatus.code
}'
  1. Find pod exec events:
cat /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="pods" and
  .objectRef.subresource=="exec"
) | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  pod: .objectRef.name,
  namespace: .objectRef.namespace
}'
  1. Find all failed requests (4xx and 5xx):
cat /var/log/kubernetes/audit/audit.log | jq 'select(.responseStatus.code >= 400) | {
  user: .user.username,
  verb: .verb,
  resource: .objectRef.resource,
  name: .objectRef.name,
  code: .responseStatus.code
}' | head -20
  1. Find all delete operations:
cat /var/log/kubernetes/audit/audit.log | jq 'select(.verb=="delete") | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  resource: .objectRef.resource,
  name: .objectRef.name,
  namespace: .objectRef.namespace
}'
  1. Count events by verb:
cat /var/log/kubernetes/audit/audit.log | jq -r '.verb' | sort | uniq -c | sort -rn
  1. Count events by user:
cat /var/log/kubernetes/audit/audit.log | jq -r '.user.username' | sort | uniq -c | sort -rn | head -10

Expected Output

You should see a mix of events from your manual operations (kubernetes-admin user) and system components. The Secret events should appear at Request level (with request body), while Pod events at Metadata level (no body).

Exercise 5: Add a Rule to Audit RBAC Changes at RequestResponse Level

Objective: Modify the audit policy to capture RBAC changes at RequestResponse level and verify the new rule works.

Steps

  1. Edit the audit policy file:
sudo vi /etc/kubernetes/audit/audit-policy.yaml
  1. Add the RBAC rule before the catch-all rule. The updated policy should look like:
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
omitManagedFields: true
rules:
  # Don't log health check endpoints
  - level: None
    nonResourceURLs:
      - "/healthz*"
      - "/readyz*"
      - "/livez*"

  # Don't log events resource
  - level: None
    resources:
      - group: ""
        resources: ["events"]

  # Log Secret access at Request level
  - level: Request
    resources:
      - group: ""
        resources: ["secrets"]

  # NEW: Log RBAC changes at RequestResponse level
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
    verbs: ["create", "update", "patch", "delete"]

  # Log Pod operations at Metadata level
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]

  # Catch-all
  - level: Metadata
  1. Restart the API server (since the policy file is read at startup):
# Touch the manifest to trigger a restart
# (kubeadm/kubelet watches for file changes)
sudo touch /etc/kubernetes/manifests/kube-apiserver.yaml

If the API server does not restart automatically, force it:

# Find the kube-apiserver container
sudo crictl ps | grep kube-apiserver

# Stop the container (kubelet will restart it)
sudo crictl stop <CONTAINER_ID>
  1. Wait for the API server to come back:
until kubectl get nodes &>/dev/null; do sleep 2; done
echo "API server is back"

Exercise 6: Verify the New Audit Rule Captures RoleBinding Creation

Objective: Create an RBAC RoleBinding and verify that the audit log captures it at RequestResponse level (including the response body).

Steps

  1. Create a new RoleBinding:
kubectl create rolebinding rbac-audit-test \
  --role=audit-test-role \
  --serviceaccount=default:default \
  -n default
  1. Find the audit event for this RoleBinding creation:
cat /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="rolebindings" and
  .objectRef.name=="rbac-audit-test" and
  .verb=="create"
)'
  1. Verify the event is at RequestResponse level (should include responseObject):
cat /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="rolebindings" and
  .objectRef.name=="rbac-audit-test" and
  .verb=="create"
) | {
  level: .level,
  user: .user.username,
  verb: .verb,
  name: .objectRef.name,
  hasRequestObject: (.requestObject != null),
  hasResponseObject: (.responseObject != null),
  responseKind: .responseObject.kind
}'

Expected Output

{
  "level": "RequestResponse",
  "user": "kubernetes-admin",
  "verb": "create",
  "name": "rbac-audit-test",
  "hasRequestObject": true,
  "hasResponseObject": true,
  "responseKind": "RoleBinding"
}

Key verification points:

  • level is RequestResponse (not Metadata)
  • hasRequestObject is true — the request body was logged
  • hasResponseObject is true — the response body was logged
  • responseObject.kind is RoleBinding — the full RoleBinding resource is in the log
  1. Inspect the full response object:
cat /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="rolebindings" and
  .objectRef.name=="rbac-audit-test" and
  .verb=="create"
) | .responseObject'

This shows the complete RoleBinding resource as returned by the API server, including server-generated fields like metadata.uid, metadata.creationTimestamp, and metadata.resourceVersion.

Cleanup

Remove the test resources created during the lab:

# Delete test resources
kubectl delete rolebinding audit-test-binding rbac-audit-test -n default --ignore-not-found
kubectl delete role audit-test-role -n default --ignore-not-found
kubectl delete pod audit-test-pod -n default --ignore-not-found
kubectl delete secret audit-test-secret -n default --ignore-not-found

To remove audit logging entirely (not recommended — keep it for security):

# Remove audit flags from kube-apiserver.yaml
# Remove the volumeMounts and volumes for audit-policy and audit-log
# The API server will restart without audit logging

# Clean up log files
sudo rm -rf /var/log/kubernetes/audit/

# Optionally remove the policy file
sudo rm -rf /etc/kubernetes/audit/

Lab Summary

Exercise What You Practiced Key Skill
1 Writing an audit policy YAML Policy structure, rule ordering, audit levels
2 Configuring kube-apiserver Manifest editing, volumeMounts, volumes, flag configuration
3 Generating audit events Creating secrets, pods, exec, RBAC resources
4 Analyzing logs with jq Filtering by resource, verb, user, status code
5 Adding a new audit rule Policy modification, API server restart
6 Verifying RequestResponse level Confirming request/response bodies are captured

Practice Questions

Question 1

You configured audit logging but the audit log file is empty. What are the most likely causes?

Show Answer

Most likely causes:

  1. The API server did not restart with the new flags. Verify with: 
    ps aux | grep kube-apiserver | grep audit-policy-file

    If the flag is not present, the manifest change was not picked up. Try:

    sudo crictl ps | grep kube-apiserver
# If the container start time is old, force a restart

  2. The volumeMount path does not match the flag path. If --audit-log-path=/var/log/kubernetes/audit/audit.log but the volumeMount is at /var/log/audit, the API server writes to a path inside the container that is not persisted to the host.

  3. The policy file has only None level rules. If every rule is level: None, all events are suppressed and nothing is written.

  4. The log directory does not exist or lacks write permissions. Verify: 
    ls -la /var/log/kubernetes/audit/
  5. The API server crashed and reverted to the old manifest. Check: 
    sudo journalctl -u kubelet --since "5 minutes ago" | grep apiserver

Question 2

Write the complete set of volumeMounts and volumes needed in the kube-apiserver manifest for audit logging.

Show Answer

volumeMounts (inside spec.containers[0].volumeMounts):

    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log

volumes (inside spec.volumes):

  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log

Key details:

  • The policy volume is readOnly: true because the API server only reads the policy file
  • The log volume is NOT readOnly because the API server must write to the audit log file
  • type: DirectoryOrCreate ensures the directory is created if it does not exist
  • The volume names (audit-policy, audit-log) must match between the volumeMount and volume definitions

Question 3

After adding a new rule to the audit policy, do you need to restart the API server? How?

Show Answer

Yes, you must restart the API server. The audit policy file is read once at API server startup. Changes to the policy file are not automatically detected.

Methods to restart the API server:

  1. Touch the manifest file (triggers kubelet to detect a change): 
    sudo touch /etc/kubernetes/manifests/kube-apiserver.yaml

    Note: This may not always trigger a restart if the file content has not changed.

  2. Make a trivial change to the manifest (e.g., add/remove a blank line): 
    sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
# Add a comment or blank line, save
  3. Stop the API server container directly: 
    sudo crictl ps | grep kube-apiserver
sudo crictl stop <CONTAINER_ID>

    The kubelet will restart it with the same manifest, but the API server will re-read the policy file.

  4. Move the manifest out and back: 
    sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sleep 5
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

Wait for the API server to come back:

until kubectl get nodes &>/dev/null; do sleep 2; done

Question 4

How do you verify that a specific audit rule is working correctly after configuration?

Show Answer

Verification steps:

  1. Identify the target resource and level. For example, if your rule logs RBAC changes at RequestResponse: ```yaml
    • level: RequestResponse resources:
      • group: “rbac.authorization.k8s.io” resources: [“rolebindings”] verbs: [“create”] ```
  2. Generate a matching event: 
    kubectl create rolebinding test-verify --role=edit --serviceaccount=default:default -n default
  3. Find the event in the audit log: 
    cat /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="rolebindings" and
  .objectRef.name=="test-verify" and
  .verb=="create"
) | {level: .level, hasRequestObject: (.requestObject != null), hasResponseObject: (.responseObject != null)}'
  4. Verify the level matches your rule:
    • Metadata → requestObject and responseObject should be null
    • Request → requestObject should be non-null, responseObject should be null
    • RequestResponse → both requestObject and responseObject should be non-null
  5. Clean up: 
    kubectl delete rolebinding test-verify -n default

Question 5

In a CKS exam scenario, you are asked to configure audit logging so that all requests to create or delete Secrets in any namespace are logged at RequestResponse level, and all other requests are logged at Metadata level. Write the complete audit policy and the API server flags needed.

Show Answer

Audit policy (/etc/kubernetes/audit/audit-policy.yaml):

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
omitManagedFields: true
rules:
  - level: None
    nonResourceURLs:
      - "/healthz*"
      - "/readyz*"
      - "/livez*"

  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["create", "delete"]

  - level: Metadata

API server flags:

--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100

Required volumeMounts:

- mountPath: /etc/kubernetes/audit
  name: audit-policy
  readOnly: true
- mountPath: /var/log/kubernetes/audit
  name: audit-log

Required volumes:

- hostPath:
    path: /etc/kubernetes/audit
    type: DirectoryOrCreate
  name: audit-policy
- hostPath:
    path: /var/log/kubernetes/audit
    type: DirectoryOrCreate
  name: audit-log

Key points:

  • The RequestResponse rule for secrets must come before the Metadata catch-all (first match wins)
  • The verbs filter restricts RequestResponse logging to only create and delete — get, list, and watch will fall through to the Metadata catch-all
  • The None rule for health endpoints prevents unnecessary noise
Previous Up Next
Audit Best Practices CKS Home Audit Logging

Updated:

You may also enjoy