CKS — Audit Best Practices

11 minute read

Audit Best Practices

CKS Exam Domain: Domain 6 — Monitoring, Logging & Runtime Security (20%)

A well-designed audit logging strategy balances three competing concerns: security visibility (log enough to detect threats), performance (don’t overload the API server), and cost (store and process audit data efficiently). This section covers production-grade audit policies, log management, compliance requirements, and the audit policy template you should know for the CKS exam.

1. Balancing Audit Detail vs Performance

The audit level you choose has a direct impact on API server performance and log volume.

Performance Impact by Level

Level Payload Size Disk I/O CPU Overhead Use When
None 0 bytes None None Health checks, discovery endpoints, system component reads
Metadata ~500 bytes–1 KB Low Minimal General activity tracking, catch-all rule
Request 1–50 KB Medium Low–Medium Secrets, configmaps, RBAC changes
RequestResponse 5–500+ KB High Medium–High Forensic-critical resources (exec, token creation)

The RequestResponse Trap

RequestResponse logs both the request body and response body. For list operations, the response body can be megabytes — every Pod, every Secret, every Deployment in the namespace. A single kubectl get pods -A at RequestResponse level generates an audit event that can be 10+ MB.

Rule: Never use RequestResponse as a catch-all. Reserve it for:

  • pods/exec and pods/attach (small payloads, high security value)
  • RBAC mutations (role/binding creation — small payloads)
  • TokenReview and CertificateSigningRequest creation

Reducing Log Volume

# 1. Omit RequestReceived stage (reduces events by ~50%)
omitStages:
  - "RequestReceived"

# 2. Omit managedFields (reduces event size by 30-60%)
omitManagedFields: true

# 3. Exclude system component reads
- level: None
  users:
    - "system:kube-controller-manager"
    - "system:kube-scheduler"
    - "system:kube-proxy"
    - "system:apiserver"
  verbs: ["get", "list", "watch"]

# 4. Exclude health endpoints
- level: None
  nonResourceURLs:
    - "/healthz*"
    - "/readyz*"
    - "/livez*"
    - "/metrics"

A layered audit policy follows the principle of specific rules first, catch-all last:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
omitManagedFields: true
rules:
  # ─── Layer 1: Exclude noise ───────────────────────────
  # Health and discovery endpoints
  - level: None
    nonResourceURLs:
      - "/healthz*"
      - "/readyz*"
      - "/livez*"
      - "/metrics"
      - "/api"
      - "/api/v1"
      - "/apis"
      - "/apis/*"
      - "/version"
      - "/openapi/*"
      - "/.well-known/*"

  # System component read operations
  - level: None
    users:
      - "system:apiserver"
      - "system:kube-controller-manager"
      - "system:kube-scheduler"
      - "system:kube-proxy"
    verbs: ["get", "list", "watch"]

  # Kubelet node status updates (very high frequency)
  - level: None
    users:
      - "system:node:*"
    resources:
      - group: ""
        resources: ["nodes/status"]
    verbs: ["patch", "update"]

  # Events are informational, not security-relevant
  - level: None
    resources:
      - group: ""
        resources: ["events"]

  # ─── Layer 2: High-security resources ─────────────────
  # Secrets — log at Request level (captures secret values)
  - level: Request
    resources:
      - group: ""
        resources: ["secrets"]

  # ConfigMaps — log at Request level
  - level: Request
    resources:
      - group: ""
        resources: ["configmaps"]

  # Pod exec/attach/portforward — log at RequestResponse
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # RBAC mutations — log at RequestResponse
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
    verbs: ["create", "update", "patch", "delete"]

  # Authentication tokens and certificates
  - level: RequestResponse
    resources:
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
      - group: "certificates.k8s.io"
        resources: ["certificatesigningrequests"]
      - group: ""
        resources: ["serviceaccounts/token"]

  # ─── Layer 3: Medium-security resources ───────────────
  # Pod/Deployment/StatefulSet mutations
  - level: Request
    resources:
      - group: ""
        resources: ["pods"]
      - group: "apps"
        resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["create", "update", "patch", "delete"]

  # Namespace operations
  - level: Request
    resources:
      - group: ""
        resources: ["namespaces"]
    verbs: ["create", "update", "patch", "delete"]

  # Node operations
  - level: Request
    resources:
      - group: ""
        resources: ["nodes"]
    verbs: ["create", "update", "patch", "delete"]

  # ─── Layer 4: Catch-all ──────────────────────────────
  # Everything else at Metadata level
  - level: Metadata

3. Retaining and Rotating Audit Logs

Log Rotation Settings

Flag Recommended Value Rationale
--audit-log-maxsize 100 (MB) Rotate before files become too large to process
--audit-log-maxbackup 10 Keep 10 rotated files (= 1 GB total max)
--audit-log-maxage 30 Delete rotated files older than 30 days

Disk Space Calculation

Max disk usage = maxsize × (maxbackup + 1)
               = 100 MB × 11
               = 1.1 GB

With maxage = 30 days:
  Files older than 30 days are deleted regardless of maxbackup count.

Monitoring Disk Usage

# Check current audit log size
du -sh /var/log/kubernetes/audit/

# Count rotated files
ls -la /var/log/kubernetes/audit/

# Set up a cron job to alert if disk usage exceeds threshold
cat <<'EOF' > /etc/cron.d/audit-disk-check
*/30 * * * * root [ $(du -sm /var/log/kubernetes/audit/ | cut -f1) -gt 800 ] && echo "Audit log disk usage critical" | logger
EOF

What Happens When Disk Fills Up

If the audit log directory runs out of disk space:

  1. The API server cannot write new audit events
  2. Depending on configuration, the API server may reject requests or crash
  3. Audit events are permanently lost during this period

Mitigation: Always monitor disk usage and set appropriate rotation values.

4. Centralized Log Collection

In production, audit logs should be shipped to a centralized system for aggregation, search, and alerting across all clusters.

Architecture: Fluent Bit → Elasticsearch → Kibana

┌──────────────────────┐     ┌──────────────────────┐
│   Control Plane Node  │     │   Control Plane Node  │
│                       │     │   (HA Cluster)        │
│  /var/log/kubernetes/ │     │  /var/log/kubernetes/ │
│   audit/audit.log     │     │   audit/audit.log     │
└──────────┬───────────┘     └──────────┬───────────┘
           │                            │
           ▼                            ▼
┌──────────────────────────────────────────────────────┐
│                    Fluent Bit                          │
│  input: tail /var/log/kubernetes/audit/audit.log      │
│  parser: json                                         │
│  filter: add cluster_name tag                         │
│  output: elasticsearch                                │
└──────────────────────┬───────────────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────────────┐
│                  Elasticsearch                        │
│   Index: kubernetes-audit-YYYY.MM.DD                  │
│   Retention: 90 days (ILM policy)                     │
└──────────────────────┬───────────────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────────────┐
│                    Kibana                              │
│   Dashboards: API activity, secret access, exec       │
│   Alerts: anonymous access, RBAC changes, 403 spikes  │
└──────────────────────────────────────────────────────┘

Fluent Bit Configuration for Audit Logs

[INPUT]
    Name              tail
    Path              /var/log/kubernetes/audit/audit.log
    Parser            json
    Tag               k8s.audit
    Refresh_Interval  5
    DB                /var/log/fluent-bit/audit-position.db

[FILTER]
    Name              modify
    Match             k8s.audit
    Add               cluster_name production-us-east-1

[OUTPUT]
    Name              es
    Match             k8s.audit
    Host              elasticsearch.logging.svc.cluster.local
    Port              9200
    Index             kubernetes-audit
    Type              _doc
    Logstash_Format   On
    Logstash_Prefix   kubernetes-audit
    Retry_Limit       5

Alternative: Webhook Backend to Fluentd

Instead of reading the log file, you can use the webhook backend to push events directly:

# API server flags
--audit-webhook-config-file=/etc/kubernetes/audit/webhook-kubeconfig.yaml
--audit-webhook-batch-max-wait=5s

This avoids the need for a file-based log collector on the control plane node.

5. Compliance Requirements

Different compliance frameworks have specific audit trail requirements.

Compliance Mapping

Framework Audit Requirement Kubernetes Implementation
SOC 2 Log all access to sensitive data Audit secrets and configmaps at Request level
SOC 2 Log authentication events Audit TokenReview, CSR at RequestResponse level
SOC 2 Retain logs for 1 year Set maxage appropriately or use centralized storage with 1-year retention
HIPAA Track all access to PHI Audit all namespace-scoped resources in PHI-containing namespaces
HIPAA Access logs must be immutable Ship to immutable storage (S3 with Object Lock, WORM storage)
PCI-DSS Log all access to cardholder data Audit secrets in PCI-scoped namespaces at Request level
PCI-DSS Log all administrative actions Audit RBAC, node operations, namespace changes at RequestResponse
PCI-DSS Retain logs for 1 year (3 months immediately available) Centralized storage with tiered retention

Immutable Audit Storage

For compliance, audit logs must not be tampered with after collection:

# Ship to S3 with Object Lock (immutable)
aws s3api put-object-lock-configuration \
  --bucket kubernetes-audit-logs \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Days": 365
      }
    }
  }'

6. What to Audit at Minimum

Even with limited resources, these resources and actions must always be audited:

Resource Level Rationale
Secrets (all verbs) Request Credential access — most critical resource for attackers
ConfigMaps (write verbs) Request Configuration tampering — can inject malicious configs
RBAC resources (write verbs) RequestResponse Privilege escalation — new bindings grant access
pods/exec, pods/attach RequestResponse Lateral movement — live access to running containers
Nodes (write verbs) Request Node compromise — tainting, draining, or modifying nodes
Authentication resources RequestResponse Token creation, CSR approval — credential generation
Namespaces (write verbs) Request Namespace creation/deletion affects isolation boundaries
ServiceAccount tokens Request Token generation — new credentials for API access

The “Must-Have” Minimal Policy

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
omitManagedFields: true
rules:
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*"]
  - level: None
    resources:
      - group: ""
        resources: ["events"]
  - level: Request
    resources:
      - group: ""
        resources: ["secrets", "configmaps", "serviceaccounts/token"]
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach"]
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
    verbs: ["create", "update", "patch", "delete"]
  - level: Metadata

7. Audit Policy Template for CKS Exam Preparation

This is the template you should memorize for the CKS exam. It covers the most commonly tested scenarios:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
omitManagedFields: true
rules:
  # 1. Silence noise
  - level: None
    nonResourceURLs:
      - "/healthz*"
      - "/readyz*"
      - "/livez*"

  # 2. Secrets — ALWAYS log at Request level
  - level: Request
    resources:
      - group: ""
        resources: ["secrets"]

  # 3. Pod exec — log with full details
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # 4. RBAC changes — capture full details
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
    verbs: ["create", "update", "patch", "delete"]

  # 5. Everything else — metadata only
  - level: Metadata

Exam Workflow

  1. Create the policy file: /etc/kubernetes/audit/audit-policy.yaml
  2. Add API server flags:
    • --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    • --audit-log-path=/var/log/kubernetes/audit/audit.log
    • --audit-log-maxage=30
    • --audit-log-maxbackup=10
    • --audit-log-maxsize=100
  3. Add volumeMounts and volumes for both the policy directory and log directory
  4. Wait for restart and verify with tail /var/log/kubernetes/audit/audit.log | jq .

8. Practice Questions

Question 1

Why should you avoid using RequestResponse level as a catch-all rule?

Show Answer

RequestResponse logs the full request body AND the full response body for every matching event. When used as a catch-all:

  1. Massive log volume — list operations return the entire resource collection in the response body. A kubectl get pods -A response can be megabytes. With hundreds of controllers doing list/watch operations, disk fills rapidly.

  2. API server performance degradation — serializing and writing large response bodies adds CPU and I/O overhead to every API request.

  3. Sensitive data exposure — response bodies for list operations include ALL resources, including secrets and configmaps you may not intend to log.

  4. Cost — storing and processing 100x more data than necessary increases infrastructure costs.

Best practice: Use RequestResponse only for high-value, low-volume resources: pods/exec, RBAC mutations, authentication tokens. Use Metadata as the catch-all.

Question 2

Design an audit policy for a cluster that must comply with PCI-DSS. What resources must be audited and at what levels?

Show Answer

PCI-DSS requires logging all access to cardholder data and all administrative actions:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
omitManagedFields: true
rules:
  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*"]
  - level: None
    resources:
      - group: ""
        resources: ["events"]

  # PCI-DSS: Log all access to secrets (cardholder data may be stored here)
  - level: Request
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]

  # PCI-DSS: Log all administrative actions with full detail
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
      - group: ""
        resources: ["pods/exec", "pods/attach", "serviceaccounts/token"]
      - group: "certificates.k8s.io"
        resources: ["certificatesigningrequests"]

  # PCI-DSS: Log namespace and node changes
  - level: Request
    resources:
      - group: ""
        resources: ["namespaces", "nodes"]
    verbs: ["create", "update", "patch", "delete"]

  # Catch-all at Metadata
  - level: Metadata

Additional PCI-DSS requirements:

  • Retain logs for at least 1 year (3 months immediately available)
  • Store logs in immutable storage
  • Review logs regularly
  • Alert on suspicious activity

Question 3

What is the difference between --audit-log-maxage and --audit-log-maxbackup, and how do they interact?

Show Answer
Flag What It Does
--audit-log-maxage Deletes rotated audit log files that are older than N days
--audit-log-maxbackup Keeps only the most recent N rotated files, regardless of age

They interact as follows:

  • Both are checked — a rotated file is deleted if it violates either condition.
  • If maxage=30 and maxbackup=10, a file older than 30 days is deleted even if there are fewer than 10 backups. Similarly, if rotation happens frequently and there are 11 backup files, the oldest is deleted even if it is only 2 days old.
  • The current audit log file (audit.log) is never deleted by either setting — it is only rotated when it reaches maxsize.

Example:

maxsize=100, maxbackup=3, maxage=7

audit.log         ← current file, never deleted
audit.log.1       ← 1 day old, kept (within maxbackup and maxage)
audit.log.2       ← 3 days old, kept
audit.log.3       ← 5 days old, kept
audit.log.4       ← 8 days old, DELETED (exceeds maxage=7)

Question 4

Describe a centralized audit log collection architecture using Fluent Bit and Elasticsearch. Why is this preferred over local log files?

Show Answer

Architecture:

  1. API server writes audit events to /var/log/kubernetes/audit/audit.log on each control plane node.
  2. Fluent Bit (DaemonSet or host-level agent) tails the audit log file, parses JSON, adds cluster metadata, and forwards to Elasticsearch.
  3. Elasticsearch stores events in time-based indices (kubernetes-audit-YYYY.MM.DD) with Index Lifecycle Management (ILM) for retention.
  4. Kibana provides dashboards for visualization and alerting on suspicious patterns.

Why centralized is preferred:

Aspect Local Files Centralized
Search jq on a single node Full-text search across all clusters
Retention Limited by disk space Scalable storage with tiered retention
Alerting Manual (cron scripts) Real-time alerts (Kibana, Prometheus)
Multi-cluster No aggregation Single pane of glass
HA Lost if node fails Replicated and persistent
Compliance Hard to prove immutability Immutable indices, audit trail of access
Access control Root on node RBAC-controlled access to log system

Question 5

What are the minimum resources you must audit for security, regardless of compliance requirements?

Show Answer

At minimum, always audit:

  1. Secrets (all verbs) at Request level — credential access is the #1 attack target
  2. RBAC resources (create/update/patch/delete) at RequestResponse level — privilege escalation detection
  3. pods/exec and pods/attach at RequestResponse level — lateral movement detection
  4. Authentication resources (TokenReview, CSR) at RequestResponse level — credential generation
  5. Nodes (write verbs) at Request level — node compromise detection
  6. Namespaces (write verbs) at Request level — isolation boundary changes
  7. ServiceAccount tokens at Request level — new API credentials

Everything else should be logged at Metadata level as a catch-all, with noise sources (health endpoints, events, system component reads) excluded at None level.

This covers the MITRE ATT&CK phases most relevant to Kubernetes: credential access, privilege escalation, lateral movement, and persistence.

Previous Up Next
Monitoring API Access CKS Home Audit Logging Lab

Updated:

You may also enjoy