KCSA — Audit Logging & Monitoring
Why Audit Logging Matters
If you can’t see what’s happening in your cluster, you can’t detect attacks, investigate incidents, or prove compliance.
Kubernetes audit logging records a chronological set of activities that captures who did what, on which resources, and when. It’s the security camera system of your cluster.
Without audit logging enabled, you’re flying blind. An attacker could read Secrets, escalate privileges, or delete workloads — and you’d have no record it ever happened.
Audit Stages
Every API request passes through a series of stages. Kubernetes can generate an audit event at each stage.
| Stage | When It Fires | What It Captures |
|---|---|---|
| RequestReceived | Immediately when the API server receives the request, before it’s processed | Who is making the request and what they’re asking for — but no response yet |
| ResponseStarted | When the response headers are sent, but before the response body (only for long-running requests like watch) |
The start of the response for streaming operations |
| ResponseComplete | After the response body has been sent and the request is finished | The full request and response — the most common stage to audit |
| Panic | When the API server panics while processing the request | Internal server errors — rare but important for debugging |
For most security purposes, you care about RequestReceived (to know what was attempted) and ResponseComplete (to know what actually happened).
Audit Levels
Audit levels control how much detail is captured for each event. These form a hierarchy from least to most verbose.
| Level | What’s Recorded | Detail | Use Case |
|---|---|---|---|
| None | Nothing — the event is not logged at all | No data | Skip noisy, low-value events (health checks, metrics) |
| Metadata | Request metadata only — user, timestamp, resource, verb, response code | Low | Good balance of visibility and volume. Suitable for most resources |
| Request | Metadata + the request body | Medium | Useful for auditing writes — you see what was created/updated |
| RequestResponse | Metadata + request body + response body | High | Maximum visibility. Use selectively — generates large volumes of data |
Choose your audit level carefully. Logging everything at RequestResponse will generate enormous volumes of data and impact performance. Use Metadata as the default and Request or RequestResponse only for sensitive resources.
Audit Policy
An audit policy defines which events should be recorded and at what level. It’s configured as a YAML file passed to the API server via --audit-policy-file.
Audit Policy Structure
apiVersion: audit.k8s.io/v1
kind: Policy
# Don't log requests to the following
omitStages:
- "RequestReceived"
rules:
# Don't log watch requests by the system
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: ""
resources: ["endpoints", "services"]
# Log Secret access at the Request level
- level: Request
resources:
- group: ""
resources: ["secrets"]
# Log RBAC changes at RequestResponse level
- level: RequestResponse
resources:
- group: "rbac.authorization.k8s.io"
resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
# Log Pod exec and attach at RequestResponse level
- level: RequestResponse
resources:
- group: ""
resources: ["pods/exec", "pods/attach"]
# Log everything else at Metadata level
- level: Metadata
Policy Rule Matching
Rules are evaluated in order — the first matching rule wins. This is why you should put the most specific rules first and the catch-all rule last.
Each rule can match on:
| Field | Purpose |
|---|---|
level |
The audit level for matching requests |
users |
List of users to match |
userGroups |
List of groups to match |
verbs |
API verbs to match (get, list, create, update, delete, watch, etc.) |
resources |
API resources to match (pods, secrets, deployments, etc.) |
namespaces |
Namespaces to match |
nonResourceURLs |
Non-resource URL paths (e.g., /healthz, /version) |
Audit Backends
Once audit events are generated, they need to go somewhere. Kubernetes supports two backends.
Log File Backend
Writes audit events to a local JSON file on the API server node.
| API Server Flag | Purpose |
|---|---|
--audit-log-path |
Path to the audit log file (e.g., /var/log/kubernetes/audit.log) |
--audit-log-maxage |
Maximum number of days to retain old audit log files |
--audit-log-maxbackup |
Maximum number of old audit log files to retain |
--audit-log-maxsize |
Maximum size in megabytes of the audit log file before it gets rotated |
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100
Webhook Backend
Sends audit events to an external HTTP endpoint (e.g., a SIEM, logging service, or custom receiver).
| API Server Flag | Purpose |
|---|---|
--audit-webhook-config-file |
Path to a kubeconfig-formatted file specifying the webhook endpoint |
--audit-webhook-batch-max-size |
Maximum number of events in a batch |
--audit-webhook-batch-max-wait |
Maximum time to wait before sending a batch |
The webhook backend is used when you want audit events to flow directly into a centralized logging system without relying on log file collection.
Comparison
| Aspect | Log File | Webhook |
|---|---|---|
| Destination | Local file on the API server node | External HTTP endpoint |
| Latency | Fast — local disk write | Depends on network and receiver |
| Reliability | Logs persist even if the receiver is down | Events may be lost if the webhook endpoint is unreachable |
| Scalability | Limited by disk space | Scales with the receiving system |
| Integration | Requires log collection agent (Fluentd, Filebeat) | Direct integration with SIEM/logging tools |
| Use case | Development, small clusters, compliance archival | Production clusters with centralized logging |
Security-Relevant Events to Audit
Not all events are equally important for security. Focus your audit policy on the actions most likely to indicate an attack or policy violation.
| Event | Why It Matters |
|---|---|
| Secret access (get, list, watch) | An attacker’s first move is often to read secrets for lateral movement |
| RBAC changes (create/update/delete on roles, bindings) | Privilege escalation — creating new powerful bindings |
| Pod exec / attach | Remote command execution inside containers — common attack technique |
| Service account token creation | Creating tokens that can be used to authenticate to the API server |
| Namespace creation/deletion | Could indicate unauthorized resource provisioning |
| Node operations | Unauthorized node registration or modification |
| Admission webhook changes | Modifying admission webhooks could bypass security policies |
| Privileged Pod creation | Creating Pods with privileged: true or host-level access |
A good audit policy captures all of these at Request or RequestResponse level while keeping routine read operations at Metadata level.
Integration with SIEM and Logging Tools
Audit logs are only useful if they’re collected, indexed, and searchable. In production, audit events flow into centralized logging and security information and event management (SIEM) systems.
| Tool | Type | Integration Method |
|---|---|---|
| Elasticsearch + Kibana | Log aggregation + visualization | Collect audit log files with Fluentd/Filebeat, index in Elasticsearch, visualize in Kibana |
| Splunk | SIEM | Use the webhook backend or a Splunk forwarder to ingest audit events |
| Datadog | Monitoring + SIEM | Datadog agent collects audit logs. Native Kubernetes integration |
| Grafana Loki | Log aggregation | Lightweight log collection with Promtail. Good for cost-effective logging |
| AWS CloudTrail | Cloud audit | EKS control plane logs can be sent to CloudTrail for unified cloud + K8s auditing |
| Azure Monitor | Cloud audit | AKS sends audit logs to Azure Monitor / Log Analytics |
| GCP Cloud Logging | Cloud audit | GKE integrates natively with Cloud Logging for audit events |
Monitoring for Security
Audit logging captures what happened. Monitoring tells you what’s happening right now.
Key Monitoring Components
| Component | Purpose | Security Use |
|---|---|---|
| metrics-server | Provides resource usage metrics (CPU, memory) for nodes and Pods | Detect resource abuse, cryptomining |
| Prometheus | Time-series metrics collection and alerting | Alert on anomalous API server request rates, authentication failures, error codes |
| Grafana | Visualization and dashboarding | Security dashboards showing RBAC denials, audit event trends, resource utilization |
| Falco | Runtime security monitoring | Detects unexpected behavior — shell access in containers, sensitive file reads, privilege escalation |
| kube-bench | CIS Benchmark compliance checking | Validates cluster configuration against the CIS Kubernetes Benchmark |
Falco is worth extra attention for the KCSA exam. It’s a CNCF graduated project that monitors system calls at runtime and can detect threats like:
- A shell being opened inside a container
- A process reading
/etc/shadowor/etc/passwd - Unexpected network connections
- Modifications to sensitive paths
Security Monitoring Best Practices
| Practice | Description |
|---|---|
| Alert on authentication failures | Spikes in 401/403 responses may indicate brute-force or credential stuffing |
| Monitor RBAC denials | Track Forbidden responses to detect misconfiguration or unauthorized access attempts |
| Track API server latency | Unusual latency spikes could indicate a denial-of-service attack |
| Monitor etcd health | etcd availability directly affects cluster health. Watch for leader elections, high latency |
| Alert on privileged Pod creation | Anything running as privileged should trigger an alert |
| Baseline normal behavior | Understand what normal API traffic looks like so deviations are obvious |
Key Takeaways for the Exam
| Concept | What to Remember |
|---|---|
| Audit stages | RequestReceived, ResponseStarted, ResponseComplete, Panic |
| Audit levels | None < Metadata < Request < RequestResponse (hierarchy of detail) |
| Policy rules | Evaluated in order — first match wins. Put specific rules first, catch-all last |
| Log file backend | --audit-log-path — writes JSON to a local file |
| Webhook backend | --audit-webhook-config-file — sends events to an external endpoint |
| Critical events | Secret access, RBAC changes, pod/exec, privileged Pod creation |
| SIEM integration | Audit logs flow into Elasticsearch, Splunk, Datadog, or cloud-native logging |
| Falco | CNCF runtime security tool — monitors syscalls for unexpected behavior |
| No audit = no visibility | Without audit logging, you cannot detect, investigate, or prove anything |
| Performance impact | RequestResponse on everything is expensive. Be selective with audit levels |
Audit logging is your detective control — it won’t prevent an attack, but it ensures you’ll know it happened. Enable it in every production cluster.
