CKS — API Server Audit Configuration

9 minute read

API Server Audit Configuration

Kubernetes audit logging records every request to the API server — who did what, to which resource, when, and with what result. It is essential for security monitoring, incident response, and compliance. The CKS exam frequently tests audit policy configuration.

Why Audit Logging Matters

Without audit logs, you cannot:

  • Detect unauthorized API access
  • Investigate security incidents (who deleted that Pod? who read those Secrets?)
  • Satisfy compliance requirements (SOC2, PCI-DSS, HIPAA)
  • Monitor for suspicious patterns (repeated failed auth attempts, privilege escalation)

Audit Policy Levels

The audit policy defines what to record for each API request. There are four levels, each capturing progressively more data:

Level What Is Recorded Use Case
None Nothing — request is not logged Exclude noisy, low-value endpoints (e.g., health checks)
Metadata Request metadata: user, timestamp, resource, verb, response code General monitoring — lightweight, low storage
Request Metadata + request body Record what was submitted (e.g., Pod spec, Secret data)
RequestResponse Metadata + request body + response body Full forensic detail — high storage cost

Storage Impact

None            → 0 bytes per request
Metadata        → ~500 bytes per request
Request         → ~2-10 KB per request (depends on body size)
RequestResponse → ~5-50 KB per request (doubles with response)

CKS Exam Tip: Use RequestResponse only for security-critical resources (Secrets, RBAC). Use Metadata for general coverage. Use None for noisy endpoints.

Audit Policy Structure

The audit policy is a YAML file that defines an ordered list of rules. The first matching rule wins — order matters.

Basic Structure

apiVersion: audit.k8s.io/v1
kind: Policy
# Exclude requests to certain non-resource URLs
omitStages:
  - "RequestReceived"    # Optional: skip logging at this stage
rules:
  # Rule 1: Most specific rules first
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets"]

  # Rule 2: Less specific
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods"]

  # Rule 3: Catch-all (last rule)
  - level: Metadata

Rule Fields

Field Description Example
level Audit level for matching requests None, Metadata, Request, RequestResponse
resources API resources to match [{group: "", resources: ["secrets"]}]
namespaces Restrict to specific namespaces ["kube-system", "production"]
verbs HTTP verbs to match ["create", "update", "delete"]
users User identities to match ["system:anonymous"]
userGroups User groups to match ["system:authenticated"]
nonResourceURLs Non-resource URLs (/healthz, /metrics) ["/healthz*", "/version"]
omitStages Stages to skip for this rule ["RequestReceived"]

Audit Stages

Each API request passes through up to four stages. The audit policy can log at each stage:

Stage When Description
RequestReceived Request arrives Before it’s delegated to handler. Logged immediately but may not have response info.
ResponseStarted Response headers sent Only for long-running requests (watch). Body not yet fully sent.
ResponseComplete Response body sent Full response available. Most commonly used.
Panic A panic occurred Internal server error. Always log. 
# Skip RequestReceived to reduce duplicate entries
# (most events will be logged at ResponseComplete anyway)
omitStages:
  - "RequestReceived"

Complete Audit Policy Example

This is a production-grade audit policy suitable for the CKS exam:

apiVersion: audit.k8s.io/v1
kind: Policy

omitStages:
  - "RequestReceived"

rules:
  # 1. Do NOT log requests to the following non-resource URLs
  - level: None
    nonResourceURLs:
    - "/healthz*"
    - "/readyz*"
    - "/livez*"
    - "/metrics"

  # 2. Do NOT log watch requests (very noisy)
  - level: None
    verbs: ["watch"]

  # 3. Do NOT log kube-system service account token fetches
  - level: None
    users:
    - "system:kube-proxy"
    - "system:kube-scheduler"
    - "system:kube-controller-manager"
    verbs: ["get"]
    resources:
    - group: ""
      resources: ["endpoints", "services", "services/status"]

  # 4. Log Secret operations at RequestResponse level (full forensics)
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets"]

  # 5. Log RBAC changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: "rbac.authorization.k8s.io"
      resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

  # 6. Log authentication-related events
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["serviceaccounts", "serviceaccounts/token"]
    - group: "authentication.k8s.io"
      resources: ["tokenreviews"]

  # 7. Log Pod creation and deletion at Request level
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
    - group: ""
      resources: ["pods"]
    - group: "apps"
      resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]

  # 8. Log exec and attach (container access) at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # 9. Log namespace operations
  - level: Metadata
    resources:
    - group: ""
      resources: ["namespaces"]

  # 10. Catch-all: log everything else at Metadata level
  - level: Metadata

Configuring the API Server for Audit Logging

Log Backend

The log backend writes audit events to a file on the control plane node.

# /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
  - command:
    - kube-apiserver
    # ... existing flags ...
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30           # Retain logs for 30 days
    - --audit-log-maxbackup=10        # Keep 10 rotated log files
    - --audit-log-maxsize=100         # Rotate after 100 MB
    volumeMounts:
    - name: audit-policy
      mountPath: /etc/kubernetes/audit
      readOnly: true
    - name: audit-logs
      mountPath: /var/log/kubernetes/audit
  volumes:
  - name: audit-policy
    hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
  - name: audit-logs
    hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate

API Server Audit Flags Reference

Flag Description Default
--audit-policy-file Path to the audit policy YAML None (audit disabled)
--audit-log-path Path for the audit log file None
--audit-log-maxage Max days to retain old log files 0 (no limit)
--audit-log-maxbackup Max number of old log files to retain 0 (no limit)
--audit-log-maxsize Max size in MB before rotation 0 (no limit)
--audit-log-format Format of audit logs (json or legacy) json
--audit-webhook-config-file Path to webhook backend config None
--audit-webhook-batch-max-wait Max time before flushing webhook batch 30s

Webhook Backend

Instead of (or in addition to) writing to a log file, you can send audit events to an external webhook.

Webhook Configuration

# /etc/kubernetes/audit/webhook-config.yaml
apiVersion: v1
kind: Config
clusters:
- name: audit-webhook
  cluster:
    server: https://audit-collector.example.com/audit
    certificate-authority: /etc/kubernetes/audit/ca.crt
contexts:
- name: audit-webhook
  context:
    cluster: audit-webhook
current-context: audit-webhook

API Server Flag

- --audit-webhook-config-file=/etc/kubernetes/audit/webhook-config.yaml
- --audit-webhook-batch-max-wait=5s

Log Backend vs. Webhook Backend

Feature Log Backend Webhook Backend
Output Local file External HTTP endpoint
Reliability High (local filesystem) Depends on network/webhook availability
Centralization Requires log shipping Built-in centralization
Latency Low Higher (network round-trip)
Typical use Simple setups, exam scenarios Production with SIEM integration

CKS Exam Tip: The exam almost always uses the log backend. Know the audit policy file path, log path, and volume mount configuration by heart.

Reading Audit Logs

Audit log entries are JSON objects. Here’s how to read and filter them:

Sample Audit Log Entry

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "abc-123-def",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/secrets",
  "verb": "list",
  "user": {
    "username": "system:serviceaccount:default:suspicious-sa",
    "groups": ["system:serviceaccounts", "system:authenticated"]
  },
  "sourceIPs": ["10.244.1.5"],
  "objectRef": {
    "resource": "secrets",
    "namespace": "default",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "code": 200
  },
  "requestReceivedTimestamp": "2025-02-09T10:00:00.000000Z",
  "stageTimestamp": "2025-02-09T10:00:00.050000Z"
}

Filtering Audit Logs

# Find all Secret access events
cat /var/log/kubernetes/audit/audit.log | \
  jq 'select(.objectRef.resource == "secrets")'

# Find events by a specific user
cat /var/log/kubernetes/audit/audit.log | \
  jq 'select(.user.username == "system:serviceaccount:default:suspicious-sa")'

# Find failed requests (non-2xx status codes)
cat /var/log/kubernetes/audit/audit.log | \
  jq 'select(.responseStatus.code >= 400)'

# Find delete operations on critical resources
cat /var/log/kubernetes/audit/audit.log | \
  jq 'select(.verb == "delete" and (.objectRef.resource == "deployments" or .objectRef.resource == "namespaces"))'

# Count events per user
cat /var/log/kubernetes/audit/audit.log | \
  jq -r '.user.username' | sort | uniq -c | sort -rn | head -20

Step-by-Step: Enabling Audit Logging on a kubeadm Cluster

This is the most common CKS exam task for this topic:

Step 1: Create the Audit Policy File

# Create the directory
mkdir -p /etc/kubernetes/audit

# Create the audit policy
cat <<'EOF' > /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: None
    nonResourceURLs:
    - "/healthz*"
    - "/readyz*"
    - "/livez*"
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets"]
  - level: Metadata
EOF

Step 2: Create the Log Directory

mkdir -p /var/log/kubernetes/audit

Step 3: Update the API Server Manifest

Add the flags and volume mounts to /etc/kubernetes/manifests/kube-apiserver.yaml:

# Add to spec.containers[0].command:
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=3
    - --audit-log-maxsize=100

# Add to spec.containers[0].volumeMounts:
    - name: audit-policy
      mountPath: /etc/kubernetes/audit
      readOnly: true
    - name: audit-logs
      mountPath: /var/log/kubernetes/audit

# Add to spec.volumes:
  - name: audit-policy
    hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
  - name: audit-logs
    hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate

Step 4: Wait for API Server to Restart

# The kubelet detects the manifest change and restarts the API server
# Watch for the Pod to come back
watch kubectl get pods -n kube-system -l component=kube-apiserver

# If the API server doesn't come back, check:
# 1. YAML syntax in the audit policy
# 2. Volume mount paths exist
# 3. File permissions
crictl ps -a | grep apiserver
crictl logs <container-id>

Step 5: Verify

# Generate some API activity
kubectl get secrets -n default

# Check the audit log
tail -5 /var/log/kubernetes/audit/audit.log | jq .

# Verify Secret access is logged at RequestResponse level
cat /var/log/kubernetes/audit/audit.log | \
  jq 'select(.objectRef.resource == "secrets") | {verb, user: .user.username, level}'

Common Mistakes

Mistake Impact Fix
Missing volume mount for audit policy API server fails to start Add hostPath volume + volumeMount
Missing volume mount for audit log path API server fails to start Add hostPath volume + volumeMount
YAML syntax error in audit policy API server fails to start Validate with python3 -c "import yaml; yaml.safe_load(open('policy.yaml'))"
Rules in wrong order Catch-all matches before specific rules Most specific rules FIRST, catch-all LAST
Logging everything at RequestResponse Disk fills up quickly Use None for health checks, Metadata as default
Forgetting omitStages Duplicate entries for each request Add omitStages: ["RequestReceived"]

Practice Questions

Q1. Write an audit policy that logs all Secret access at RequestResponse level, all Pod creation/deletion at Request level, and everything else at Metadata level. Exclude health check endpoints entirely.

Show Answer 
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # Exclude health checks
  - level: None
    nonResourceURLs:
    - "/healthz*"
    - "/readyz*"
    - "/livez*"

  # Log Secret access with full request and response
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets"]

  # Log Pod create/delete with request body
  - level: Request
    verbs: ["create", "delete"]
    resources:
    - group: ""
      resources: ["pods"]

  # Everything else at Metadata
  - level: Metadata

Q2. What API server flags are required to enable audit logging with a log backend? List them with appropriate values.

Show Answer

Minimum required:

--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log

Recommended additional flags:

--audit-log-maxage=30        # Retain logs for 30 days
--audit-log-maxbackup=10     # Keep 10 rotated files
--audit-log-maxsize=100      # Rotate at 100 MB

You must also add the corresponding volume mounts and hostPath volumes to the API server Pod spec.

Q3. The API server fails to start after you added audit logging configuration. How do you troubleshoot?

Show Answer 
# 1. Check if the API server container is crashing
crictl ps -a | grep apiserver

# 2. Check container logs
crictl logs <container-id>

# 3. Common causes:
# a) Audit policy file doesn't exist at the specified path
ls -la /etc/kubernetes/audit/audit-policy.yaml

# b) YAML syntax error in audit policy
python3 -c "import yaml; yaml.safe_load(open('/etc/kubernetes/audit/audit-policy.yaml'))"

# c) Volume mount paths don't match flag paths
grep -A5 'audit' /etc/kubernetes/manifests/kube-apiserver.yaml

# d) Log directory doesn't exist
mkdir -p /var/log/kubernetes/audit

# e) Permission issues
ls -la /var/log/kubernetes/audit/

Q4. In an audit policy, rules are evaluated in order. What happens if the first rule is - level: None with no filters?

Show Answer

A rule with level: None and no resource, verb, or user filters matches every request. Since it’s the first rule and the first match wins, nothing will be logged at all.

# BAD — this disables all auditing
rules:
  - level: None          # Matches everything — catch-all at position 1
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["secrets"]   # Never reached

Always put the most specific rules first and the catch-all last.

Q5. What are the four audit stages, and which one is most commonly used for logging?

Show Answer
Stage Description
RequestReceived Event generated as soon as the request arrives, before processing
ResponseStarted Response headers sent but body not complete (only for long-running requests like watch)
ResponseComplete Response body has been fully sent — most commonly used
Panic An internal server error (panic) occurred during request handling

ResponseComplete is the most useful because it includes the final response status code. Most policies use omitStages: ["RequestReceived"] to avoid duplicate entries, since every request logged at RequestReceived will also be logged at ResponseComplete.

Previous Up Next
Upgrade Frequency CKS Home Cluster Hardening Lab

Updated:

You may also enjoy