CKS — Audit Log Analysis

9 minute read

Audit Log Analysis

CKS Exam Domain: Domain 6 — Monitoring, Logging & Runtime Security (20%)

Collecting audit logs is only half the equation — you must be able to read and query them to detect security incidents, answer compliance questions, and investigate breaches. The CKS exam may present you with an audit log file and ask you to find specific events using jq, grep, or manual inspection.

Kubernetes audit events are JSON objects written one per line. Mastering jq is the fastest way to extract answers from large audit logs.

1. Audit Log JSON Structure

Every audit event follows the audit.k8s.io/v1 Event structure. Here is an annotated example:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Request",
  "auditID": "a]b12c34-5678-90ab-cdef-1234567890ab",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/secrets/db-password",
  "verb": "get",
  "user": {
    "username": "system:serviceaccount:default:app-sa",
    "uid": "12345-67890",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:default",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "10.244.1.15"
  ],
  "userAgent": "kubectl/v1.30.0 (linux/amd64) kubernetes/abcdef1",
  "objectRef": {
    "resource": "secrets",
    "namespace": "default",
    "name": "db-password",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2025-02-09T14:30:00.123456Z",
  "stageTimestamp": "2025-02-09T14:30:00.125789Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"app-secret-reader\" of Role \"secret-reader\" to ServiceAccount \"app-sa\" in namespace \"default\""
  }
}

Key Fields Reference

Field Description Security Relevance
user.username Who made the request Identify the actor — human, service account, or anonymous
user.groups Group membership of the requester Detect requests from unexpected groups
verb API action: get, list, watch, create, update, patch, delete Detect destructive or read operations on sensitive resources
objectRef.resource Target resource type Filter for secrets, roles, pods/exec
objectRef.namespace Target namespace Scope investigation to a namespace
objectRef.name Specific resource name Find access to a specific secret or role
responseStatus.code HTTP response status 403 = forbidden, 401 = unauthorized, 200 = success
sourceIPs Client IP address(es) Correlate with known IPs, detect external access
requestURI Full request path Identify exact API endpoint accessed
userAgent Client tool/version Detect unusual tools (curl, custom scripts vs kubectl)
requestReceivedTimestamp When the API server received the request Timeline reconstruction
stageTimestamp When the current stage completed Measure request latency
annotations Authorization decision and reason See which RBAC binding allowed/denied access

2. Filtering with jq

jq is the essential tool for audit log analysis. Each line in the audit log is separate JSON object — use jq to filter, extract fields, and aggregate.

Basic Syntax

# Read events from the audit log (each line is a JSON object)
cat /var/log/kubernetes/audit/audit.log | jq '<filter>'

# For large files, combine with grep for pre-filtering
grep '"secrets"' /var/log/kubernetes/audit/audit.log | jq '<filter>'

Filter by Verb

# Find all delete operations
cat audit.log | jq 'select(.verb=="delete")'

# Find all create and update operations
cat audit.log | jq 'select(.verb=="create" or .verb=="update")'

# Find all write operations (create, update, patch, delete)
cat audit.log | jq 'select(.verb | IN("create","update","patch","delete"))'

Filter by Resource

# Find all Secret access
cat audit.log | jq 'select(.objectRef.resource=="secrets")'

# Find all RBAC changes
cat audit.log | jq 'select(.objectRef.resource | IN("roles","rolebindings","clusterroles","clusterrolebindings"))'

# Find pod exec events
cat audit.log | jq 'select(.objectRef.resource=="pods" and .objectRef.subresource=="exec")'

Filter by Response Code

# Find all failed requests (4xx/5xx)
cat audit.log | jq 'select(.responseStatus.code >= 400)'

# Find forbidden requests (403)
cat audit.log | jq 'select(.responseStatus.code == 403)'

# Find unauthorized requests (401)
cat audit.log | jq 'select(.responseStatus.code == 401)'

Filter by User

# Find all anonymous requests
cat audit.log | jq 'select(.user.username=="system:anonymous")'

# Find requests from a specific service account
cat audit.log | jq 'select(.user.username=="system:serviceaccount:default:app-sa")'

# Find requests from users NOT in system groups
cat audit.log | jq 'select(.user.groups | all(. != "system:masters"))'

Filter by Namespace

# Find all events in the production namespace
cat audit.log | jq 'select(.objectRef.namespace=="production")'

# Find events across multiple namespaces
cat audit.log | jq 'select(.objectRef.namespace | IN("production","finance","hr"))'

Combine Multiple Filters

# Secret deletions in production namespace
cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  .verb=="delete" and
  .objectRef.namespace=="production"
)'

# Failed secret access attempts (403)
cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  .responseStatus.code==403
)'

# Pod exec by non-admin users
cat audit.log | jq 'select(
  .objectRef.resource=="pods" and
  .objectRef.subresource=="exec" and
  (.user.groups | all(. != "system:masters"))
)'

Extract Specific Fields

# Get a summary of who accessed what
cat audit.log | jq '{user: .user.username, verb: .verb, resource: .objectRef.resource, name: .objectRef.name, code: .responseStatus.code}'

# Get unique users who accessed secrets
cat audit.log | jq 'select(.objectRef.resource=="secrets") | .user.username' | sort -u

# Count events by verb
cat audit.log | jq '.verb' | sort | uniq -c | sort -rn

# Get timeline of events for a specific resource
cat audit.log | jq 'select(.objectRef.name=="db-password") | {time: .requestReceivedTimestamp, user: .user.username, verb: .verb, code: .responseStatus.code}'

3. Filtering with grep

For quick searches when jq is not available or you need fast pre-filtering on large files:

# Find events mentioning secrets
grep '"secrets"' audit.log

# Find delete operations
grep '"delete"' audit.log

# Find 403 responses
grep '"code":403' audit.log

# Find anonymous access
grep 'system:anonymous' audit.log

# Combine grep with jq for performance on large files
grep '"secrets"' audit.log | grep '"delete"' | jq '.'

# Find events by timestamp range
grep '2025-02-09T14:3' audit.log | jq '.'

4. Common Audit Log Queries

These are the queries you must know for the CKS exam and real-world security investigations.

Failed Authentication Attempts

# Find 401 Unauthorized responses
cat audit.log | jq 'select(.responseStatus.code == 401) | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  sourceIP: .sourceIPs[0],
  uri: .requestURI
}'

Secret Access Events

# All secret access (read + write)
cat audit.log | jq 'select(.objectRef.resource=="secrets") | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  verb: .verb,
  namespace: .objectRef.namespace,
  name: .objectRef.name,
  code: .responseStatus.code
}'

# Secret reads only (get, list, watch)
cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  (.verb | IN("get","list","watch"))
) | {user: .user.username, verb: .verb, name: .objectRef.name}'

Pod Exec Events

# All exec events — detects lateral movement
cat audit.log | jq 'select(
  .objectRef.resource=="pods" and
  .objectRef.subresource=="exec"
) | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  pod: .objectRef.name,
  namespace: .objectRef.namespace,
  sourceIP: .sourceIPs[0]
}'

Unauthorized Access Attempts (403 Forbidden)

# All 403 responses — someone tried to do something RBAC denied
cat audit.log | jq 'select(.responseStatus.code == 403) | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  verb: .verb,
  resource: .objectRef.resource,
  namespace: .objectRef.namespace,
  name: .objectRef.name,
  reason: .annotations["authorization.k8s.io/reason"]
}'

RBAC Changes (Privilege Escalation Detection)

# Detect role/binding modifications
cat audit.log | jq 'select(
  (.objectRef.resource | IN("roles","rolebindings","clusterroles","clusterrolebindings")) and
  (.verb | IN("create","update","patch","delete"))
) | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  verb: .verb,
  resource: .objectRef.resource,
  name: .objectRef.name
}'

Anonymous API Access

# Any request from anonymous users
cat audit.log | jq 'select(.user.username == "system:anonymous") | {
  time: .requestReceivedTimestamp,
  verb: .verb,
  uri: .requestURI,
  sourceIP: .sourceIPs[0],
  code: .responseStatus.code
}'

5. Identifying Suspicious Activity Patterns

Beyond individual queries, look for patterns that indicate an attack:

Pattern 1: Credential Harvesting

An attacker enumerating secrets across namespaces:

# Many secret list/get operations from the same user across namespaces
cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  (.verb | IN("get","list"))
) | .user.username + " → " + .objectRef.namespace' | sort | uniq -c | sort -rn

If a single user lists secrets in many namespaces, this is suspicious.

Pattern 2: Privilege Escalation

New ClusterRoleBindings granting cluster-admin:

cat audit.log | jq 'select(
  .objectRef.resource=="clusterrolebindings" and
  .verb=="create"
) | {user: .user.username, binding: .objectRef.name, time: .requestReceivedTimestamp}'

Pattern 3: Lateral Movement via Exec

Many exec sessions from a single source:

cat audit.log | jq 'select(
  .objectRef.subresource=="exec"
) | .user.username + " → " + .objectRef.namespace + "/" + .objectRef.name' | sort | uniq -c | sort -rn

Pattern 4: Brute Force Authentication

Repeated 401 responses from the same source IP:

cat audit.log | jq 'select(.responseStatus.code == 401) | .sourceIPs[0]' | sort | uniq -c | sort -rn

Pattern 5: Unusual User Agent

Requests from unexpected tools:

cat audit.log | jq '.userAgent' | sort -u

Look for curl, python-requests, or custom agents that differ from expected kubectl or controller user agents.

6. Reading Audit Events Without jq

In some exam environments, jq may not be installed. Use these alternatives:

# Pretty-print with python
cat audit.log | python3 -m json.tool

# Extract a field with grep and awk
grep '"secrets"' audit.log | grep -o '"username":"[^"]*"' | sort -u

# Use jsonpath-like extraction with python
cat audit.log | python3 -c "
import json, sys
for line in sys.stdin:
    event = json.loads(line)
    if event.get('objectRef', {}).get('resource') == 'secrets':
        print(f\"{event['user']['username']} {event['verb']} {event['objectRef'].get('name', 'N/A')}\")
"

7. Practice Questions

Question 1

Write a jq command to find all audit events where someone deleted a Secret in the production namespace.

Show Answer 
cat /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  .verb=="delete" and
  .objectRef.namespace=="production"
) | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  secret: .objectRef.name,
  sourceIP: .sourceIPs[0],
  code: .responseStatus.code
}'

This filters for events where:

  • The resource is secrets
  • The verb is delete
  • The namespace is production

And outputs a summary with the timestamp, user, secret name, source IP, and response code.

Question 2

An audit log shows a event with responseStatus.code: 403. What does this mean, and what should you investigate?

Show Answer

A 403 Forbidden response means the API server denied the request because RBAC (or another authorizer) did not grant the user permission to perform the requested action.

Investigation steps:

  1. Identify the user and action: 
    cat audit.log | jq 'select(.responseStatus.code==403) | {user: .user.username, verb: .verb, resource: .objectRef.resource}'

  2. Determine if this is expected — a service account with limited permissions will legitimately get 403 errors when accessing resources outside its scope.

  3. Check for patterns — a single 403 is normal. Hundreds of 403s from the same user across different resources suggests an attacker probing for permissions.

  4. Review the authorization annotation: 
    jq '.annotations["authorization.k8s.io/reason"]'

    This tells you which RBAC rule was evaluated and why it denied access.

  5. Cross-reference with RBAC — verify the user’s RoleBindings and ClusterRoleBindings to confirm they should not have the requested access.

Question 3

How would you find all kubectl exec events in the audit log? What resource and subresource do you filter on?

Show Answer

Filter on objectRef.resource == "pods" and objectRef.subresource == "exec":

cat audit.log | jq 'select(
  .objectRef.resource=="pods" and
  .objectRef.subresource=="exec"
) | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  pod: .objectRef.name,
  namespace: .objectRef.namespace
}'

kubectl exec translates to a create verb on the pods/exec subresource. The requestURI looks like:

/api/v1/namespaces/default/pods/my-pod/exec?command=sh&stdin=true&stdout=true&tty=true

Note: you can also check the verb field — it will be create for exec sessions, not exec. Kubernetes does not have an “exec” verb.

Question 4

What is the annotations field in an audit event, and why is authorization.k8s.io/decision valuable?

Show Answer

The annotations field contains additional metadata added by the API server during request processing. The two most important audit annotations are:

  • authorization.k8s.io/decision — either allow or forbid, indicating whether the request was authorized.
  • authorization.k8s.io/reason — a human-readable explanation of why the request was allowed or denied, typically referencing the specific RBAC Role/ClusterRole and RoleBinding/ClusterRoleBinding.

Example:

"annotations": {
  "authorization.k8s.io/decision": "allow",
  "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"cluster-admin-binding\" of ClusterRole \"cluster-admin\" to User \"admin@example.com\""
}

This is invaluable for security investigations because it tells you exactly which RBAC binding authorized the action — helping you trace overly permissive bindings and confirm whether access was legitimate.

Question 5

You are given an audit log and asked: “Which service accounts accessed secrets in the last hour?” Write the command.

Show Answer 
cat /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  (.user.username | startswith("system:serviceaccount:")) and
  (.requestReceivedTimestamp > "2025-02-09T13:30:00Z")
) | .user.username' | sort -u

Breakdown:

  • .objectRef.resource=="secrets" — only secret access events
  • .user.username | startswith("system:serviceaccount:") — only service accounts (their usernames always start with system:serviceaccount:<namespace>:<name>)
  • .requestReceivedTimestamp > "2025-02-09T13:30:00Z" — events from the last hour (adjust timestamp accordingly). JSON string comparison works for ISO 8601 timestamps.
  • sort -u — deduplicate to get unique service account names

On the exam, adjust the timestamp to match the current time. You can find the current time with date -u +"%Y-%m-%dT%H:%M:%SZ" and then subtract one hour.

Previous Up Next
Audit Backends CKS Home Monitoring API Access

Updated:

You may also enjoy