CKS — Audit Backends

10 minute read

Audit Backends

CKS Exam Domain: Domain 6 — Monitoring, Logging & Runtime Security (20%)

An audit policy defines what to log. An audit backend defines where the events go. Kubernetes supports two backend types: log (writes to a file on the API server node) and webhook (sends events to an external HTTP endpoint). The CKS exam focuses heavily on the log backend because you configure it directly in the kube-apiserver static Pod manifest.

1. Log Backend Configuration

The log backend writes audit events as JSON lines to a file on the API server node. It is configured entirely through kube-apiserver flags.

Required Flags

Flag Description Example Value
--audit-policy-file Path to the audit policy YAML file /etc/kubernetes/audit/audit-policy.yaml
--audit-log-path Path where audit log files are written /var/log/kubernetes/audit/audit.log
--audit-log-maxage Maximum number of days to retain old audit log files 30
--audit-log-maxbackup Maximum number of old audit log files to retain 10
--audit-log-maxsize Maximum size in megabytes of the audit log file before it gets rotated 100

Flag Details

# Minimum required flags to enable audit logging
--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log

# Recommended rotation settings
--audit-log-maxage=30        # Keep logs for 30 days
--audit-log-maxbackup=10     # Keep at most 10 rotated files
--audit-log-maxsize=100      # Rotate when file reaches 100 MB

Critical: Both --audit-policy-file and --audit-log-path are required. If you specify a policy file without a log path (and no webhook), the API server starts but does not log any audit events. If the policy file path is invalid, the API server fails to start.

2. Configuring the API Server Manifest

This is the core CKS exam skill — modifying /etc/kubernetes/manifests/kube-apiserver.yaml to enable audit logging. You need three things: the flags, the volumeMounts, and the volumes.

Step 1: Create the Audit Policy File

sudo mkdir -p /etc/kubernetes/audit

Write your audit policy to /etc/kubernetes/audit/audit-policy.yaml:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
omitManagedFields: true
rules:
  - level: None
    nonResourceURLs:
      - "/healthz*"
      - "/readyz*"
      - "/livez*"

  - level: Request
    resources:
      - group: ""
        resources: ["secrets"]

  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach"]

  - level: Metadata

Step 2: Create the Log Directory

sudo mkdir -p /var/log/kubernetes/audit

Step 3: Modify kube-apiserver.yaml

Edit /etc/kubernetes/manifests/kube-apiserver.yaml. The following shows the complete sections you need to add or modify:

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    # ... existing flags ...
    # ADD these audit flags:
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    # ... remaining existing flags ...
    volumeMounts:
    # ... existing volumeMounts ...
    # ADD these two mounts:
    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log
  volumes:
  # ... existing volumes ...
  # ADD these two volumes:
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log

Step 4: Wait for API Server Restart

The kubelet watches /etc/kubernetes/manifests/ and automatically restarts the API server when the manifest changes.

# Watch the API server Pod restart
watch crictl ps | grep kube-apiserver

# Or check with kubectl (may fail briefly during restart)
kubectl get pods -n kube-system -l component=kube-apiserver

# Verify the API server is running with the new flags
ps aux | grep audit-policy-file

Exam Warning: If you make a YAML syntax error in the manifest, the API server will not restart. Always back up the manifest before editing:

cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak

Step 5: Verify Audit Events Are Being Written

# Check if the log file exists and has content
ls -la /var/log/kubernetes/audit/audit.log

# View the last few events
tail -5 /var/log/kubernetes/audit/audit.log | jq .

# Generate a test event and verify it appears
kubectl get secrets -n default
tail -1 /var/log/kubernetes/audit/audit.log | jq '.verb, .objectRef.resource'

3. Complete kube-apiserver YAML Reference

Here is a more complete view of what the manifest looks like with audit logging enabled:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.0.0.10:6443
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.0.0.10
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    # ─── Audit Logging Flags ────────────────────────────
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    image: registry.k8s.io/kube-apiserver:v1.30.0
    name: kube-apiserver
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log

4. Webhook Backend

The webhook backend sends audit events to an external HTTP endpoint in real-time. This is used for centralized logging to systems like Elasticsearch, Splunk, or a custom SIEM.

Configuration Flag

--audit-webhook-config-file=/etc/kubernetes/audit/webhook-config.yaml

Webhook Kubeconfig Format

The webhook config file uses the kubeconfig format:

apiVersion: v1
kind: Config
clusters:
  - name: audit-webhook
    cluster:
      server: https://audit-collector.example.com/audit
      certificate-authority: /etc/kubernetes/audit/ca.crt
contexts:
  - name: audit-webhook
    context:
      cluster: audit-webhook
      user: audit-webhook
current-context: audit-webhook
users:
  - name: audit-webhook
    user:
      client-certificate: /etc/kubernetes/audit/client.crt
      client-key: /etc/kubernetes/audit/client.key

Webhook-Specific Flags

Flag Description Default
--audit-webhook-config-file Path to webhook kubeconfig file (none)
--audit-webhook-initial-backoff Time to wait before retrying a failed webhook call 10s
--audit-webhook-batch-buffer-size Number of events to buffer before sending 10000
--audit-webhook-batch-max-size Maximum number of events in one webhook call 400
--audit-webhook-batch-max-wait Maximum time to wait before sending a batch 30s
--audit-webhook-batch-throttle-qps Maximum QPS to the webhook endpoint 10

Note: You can use both log and webhook backends simultaneously. The API server writes events to the log file AND sends them to the webhook endpoint.

5. Dynamic Audit Configuration (AuditSink)

The AuditSink resource allows you to configure audit webhook backends dynamically without restarting the API server. This feature has been in alpha since Kubernetes 1.13 and requires the DynamicAuditing feature gate.

apiVersion: auditregistration.k8s.io/v1alpha1
kind: AuditSink
metadata:
  name: external-sink
spec:
  policy:
    level: Metadata
    stages:
      - "ResponseComplete"
  webhook:
    throttle:
      qps: 10
      burst: 15
    clientConfig:
      url: "https://audit-collector.example.com/audit"

Exam Note: Dynamic auditing is alpha and unlikely to appear on the CKS exam. Focus on the static log backend configuration, which is the exam-relevant approach.

6. Log vs Webhook Backend Comparison

Aspect Log Backend Webhook Backend
Configuration API server flags only Flags + kubeconfig file
Output Local JSON file on the node HTTP POST to external endpoint
Latency Near-zero (local file write) Network-dependent
Reliability Depends on local disk Depends on external service availability
Rotation Built-in (maxage, maxbackup, maxsize) External system handles retention
Scalability Limited by disk space Scales with external infrastructure
Multi-cluster Requires separate log collection Native centralized collection
Exam Relevance High — expect to configure this Low — rarely tested directly
Use Together ✅ Yes — both can run simultaneously ✅ Yes

7. Performance Considerations

Audit logging adds overhead to every API request. The impact depends on the audit level and backend.

Factor Impact Mitigation
RequestResponse for all resources High CPU + disk I/O — response bodies can be large (multi-MB for list operations) Use RequestResponse only for critical resources (secrets, RBAC, exec)
High audit log volume on disk Disk fills up, API server may crash Set maxsize, maxbackup, maxage; monitor disk usage
Webhook backend latency Adds latency to API requests if webhook is slow Webhook uses async batching by default — impact is minimal unless buffer fills
omitStages not set Double the events (RequestReceived + ResponseComplete) Always set omitStages: ["RequestReceived"]
omitManagedFields not set 30-60% larger event payloads Set omitManagedFields: true

Sizing Guidelines

Cluster Size Estimated Events/Day Recommended maxsize Recommended maxbackup
Small (< 50 nodes) ~100K–500K 100 MB 10
Medium (50–200 nodes) ~500K–5M 200 MB 15
Large (200+ nodes) ~5M–50M+ 500 MB 20

8. Troubleshooting

API Server Fails to Start After Enabling Audit Logging

# Check kubelet logs for API server crash reason
journalctl -u kubelet --since "5 minutes ago" | grep apiserver

# Common causes:
# 1. Policy file YAML syntax error
sudo cat /etc/kubernetes/audit/audit-policy.yaml | python3 -c "import yaml, sys; yaml.safe_load(sys.stdin)"

# 2. Policy file path does not match the volumeMount
# Verify the volume and volumeMount paths match the flag value

# 3. Missing directory for log path
ls -la /var/log/kubernetes/audit/

# 4. Restore from backup
cp /tmp/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml

Audit Log File Is Empty

# Verify the API server is running with audit flags
ps aux | grep "[a]udit-policy-file"

# Check if the policy file has any rules
cat /etc/kubernetes/audit/audit-policy.yaml

# Generate a test event
kubectl get secrets -A

# Check the log file
tail -5 /var/log/kubernetes/audit/audit.log

9. Practice Questions

Question 1

What are the two audit backend types in Kubernetes, and which one is most likely to appear on the CKS exam?

Show Answer

The two audit backends are:

  1. Log backend — writes audit events as JSON lines to a file on the API server node. Configured with --audit-log-path and related flags.
  2. Webhook backend — sends audit events via HTTP POST to an external endpoint. Configured with --audit-webhook-config-file.

The log backend is the one most likely to appear on the CKS exam. You will be expected to:

  • Write an audit policy file
  • Add the --audit-policy-file and --audit-log-path flags to the kube-apiserver manifest
  • Add the corresponding volumeMounts and volumes for both the policy file and the log directory
  • Verify that audit events are being written

Question 2

List the five kube-apiserver flags needed to configure the log backend with rotation, and explain what each does.

Show Answer
Flag Purpose
--audit-policy-file Path to the audit policy YAML that defines which events to log and at what level
--audit-log-path Path to the file where audit events are written as JSON lines
--audit-log-maxage Maximum number of days to retain old rotated audit log files
--audit-log-maxbackup Maximum number of old rotated audit log files to keep
--audit-log-maxsize Maximum size in megabytes before the current audit log file is rotated

The first two flags (--audit-policy-file and --audit-log-path) are required. The rotation flags (maxage, maxbackup, maxsize) are optional but strongly recommended in production to prevent disk exhaustion.

Question 3

Why do you need to add both volumeMounts and volumes to the kube-apiserver manifest when enabling audit logging?

Show Answer

The kube-apiserver runs as a static Pod managed by the kubelet. Like all Pods, it has its own filesystem isolated from the host. The flags reference paths like /etc/kubernetes/audit/audit-policy.yaml and /var/log/kubernetes/audit/audit.log — but these paths must exist inside the container.

You need:

  1. Two volumes — hostPath volumes that map host directories into the Pod:
    • The audit policy directory (/etc/kubernetes/audit) so the API server can read the policy file
    • The audit log directory (/var/log/kubernetes/audit) so the API server can write log files
  2. Two volumeMounts — mount points inside the container:
    • The policy directory mounted readOnly: true (the API server only reads the policy)
    • The log directory mounted writable (the API server must write to the log file)

Without these mounts, the API server cannot access the policy file or write the log file, and it will fail to start.

Question 4

What happens if the audit policy file has a YAML syntax error and you save the kube-apiserver manifest?

Show Answer

The kubelet detects the change to /etc/kubernetes/manifests/kube-apiserver.yaml and attempts to restart the API server. The API server tries to parse the audit policy file during startup. If the policy file has a YAML syntax error:

  1. The API server fails to start
  2. The existing API server Pod is terminated
  3. kubectl commands stop working because there is no API server
  4. The kubelet continues trying to restart the Pod, but it keeps failing

Recovery steps:

# Fix the policy file
sudo vi /etc/kubernetes/audit/audit-policy.yaml

# Or restore the API server manifest from backup
cp /tmp/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml

Prevention: Always back up the manifest before editing, and validate the policy YAML syntax before modifying the API server manifest.

Question 5

Can you use both the log backend and webhook backend simultaneously? Describe a scenario where this is useful.

Show Answer

Yes, both backends can run simultaneously. When both are configured, the API server sends every audit event to both the local log file and the webhook endpoint.

A common production scenario:

  • Log backend → writes to /var/log/kubernetes/audit/audit.log on the API server node. This provides a local safety net — even if the external logging system is down, audit events are preserved on disk. Useful for immediate troubleshooting on the node.

  • Webhook backend → sends events to a centralized SIEM (e.g., Elasticsearch via Fluentd, or Splunk). This provides searchable, aggregated audit data across all clusters, long-term retention, alerting, and dashboards.

Configuration:

--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-log-maxage=7
--audit-log-maxbackup=3
--audit-log-maxsize=100
--audit-webhook-config-file=/etc/kubernetes/audit/webhook-config.yaml

Both backends use the same audit policy — the policy defines what to log, and both backends receive the same events.

Previous Up Next
Audit Policy CKS Home Audit Log Analysis

Updated:

You may also enjoy