CKS — Monitoring API Access

10 minute read

Monitoring API Access

CKS Exam Domain: Domain 6 — Monitoring, Logging & Runtime Security (20%)

Audit logs are the source of truth for API access in Kubernetes. But raw logs are only useful if you know what to look for. This section focuses on systematic monitoring — tracking access patterns, detecting anomalies, and correlating audit events with RBAC configuration to identify unauthorized or suspicious API activity.

1. Tracking Who Accessed What and When

Every API request is attributable to a specific identity. The audit log captures this in the user field.

Identity Types in Kubernetes

Identity Type user.username Format Example
Human user (certificate-based) Common Name from client certificate admin@example.com
Human user (OIDC) OIDC claim (typically email) jdoe@example.com
ServiceAccount system:serviceaccount:<namespace>:<name> system:serviceaccount:default:app-sa
Node (kubelet) system:node:<node-name> system:node:worker-01
Anonymous system:anonymous system:anonymous
API server (self) system:apiserver system:apiserver
Controllers system:kube-controller-manager system:kube-controller-manager

Building an Access Timeline

# Full timeline for a specific resource
cat audit.log | jq 'select(.objectRef.name=="db-password" and .objectRef.resource=="secrets") | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  verb: .verb,
  code: .responseStatus.code,
  sourceIP: .sourceIPs[0]
}' | jq -s 'sort_by(.time)'

Answering “Who Accessed This Secret?”

# All users who accessed a specific secret
cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  .objectRef.name=="db-password" and
  .objectRef.namespace=="production"
) | {user: .user.username, verb: .verb, time: .requestReceivedTimestamp}'

# Unique users only
cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  .objectRef.name=="db-password"
) | .user.username' | sort -u

2. Anonymous Request Detection

Anonymous requests are those made without any authentication credential. By default, Kubernetes assigns them the username system:anonymous and the group system:unauthenticated.

Why Anonymous Access Is Dangerous

Risk Attack Scenario
Information disclosure Anonymous user lists pods, services, or endpoints to map the cluster
API discovery Anonymous user accesses /api or /apis to enumerate available resources
Exploitation If RBAC binds roles to system:unauthenticated, anonymous users gain resource access
External exposure API server exposed to the internet without authentication

Detecting Anonymous Access

# All anonymous requests
cat audit.log | jq 'select(.user.username == "system:anonymous") | {
  time: .requestReceivedTimestamp,
  verb: .verb,
  uri: .requestURI,
  sourceIP: .sourceIPs[0],
  code: .responseStatus.code
}'

# Anonymous requests that SUCCEEDED (critical — means something granted access)
cat audit.log | jq 'select(
  .user.username == "system:anonymous" and
  .responseStatus.code < 400
) | {
  verb: .verb,
  uri: .requestURI,
  sourceIP: .sourceIPs[0],
  code: .responseStatus.code
}'

Verifying Anonymous Access Is Disabled

# Check if the API server allows anonymous authentication
ps aux | grep kube-apiserver | grep anonymous-auth

# In the API server manifest:
# --anonymous-auth=false  ← Disables anonymous access
# Default is true — anonymous requests are allowed but get minimal RBAC

Checking for Dangerous RBAC Bindings to Anonymous

# Find ClusterRoleBindings that grant access to system:unauthenticated
kubectl get clusterrolebindings -o json | jq '.items[] | select(
  .subjects[]? | select(
    .kind=="Group" and .name=="system:unauthenticated"
  )
) | {name: .metadata.name, role: .roleRef.name}'

# Find ClusterRoleBindings to system:anonymous
kubectl get clusterrolebindings -o json | jq '.items[] | select(
  .subjects[]? | select(
    .kind=="User" and .name=="system:anonymous"
  )
) | {name: .metadata.name, role: .roleRef.name}'

3. ServiceAccount Usage Monitoring

ServiceAccounts are the primary API identity for workloads running inside the cluster. Monitoring their activity is critical because a compromised Pod uses its mounted ServiceAccount token for all API access.

Listing Active ServiceAccounts

# Which ServiceAccounts are making API calls?
cat audit.log | jq 'select(
  .user.username | startswith("system:serviceaccount:")
) | .user.username' | sort | uniq -c | sort -rn | head -20

Detecting Unusual ServiceAccount Activity

# ServiceAccounts accessing secrets (potential credential theft)
cat audit.log | jq 'select(
  (.user.username | startswith("system:serviceaccount:")) and
  .objectRef.resource=="secrets"
) | {
  sa: .user.username,
  verb: .verb,
  secret: .objectRef.name,
  namespace: .objectRef.namespace
}'

# ServiceAccounts making RBAC changes (privilege escalation)
cat audit.log | jq 'select(
  (.user.username | startswith("system:serviceaccount:")) and
  (.objectRef.resource | IN("roles","rolebindings","clusterroles","clusterrolebindings")) and
  (.verb | IN("create","update","patch","delete"))
) | {
  sa: .user.username,
  verb: .verb,
  resource: .objectRef.resource,
  name: .objectRef.name
}'

# ServiceAccounts using exec (lateral movement from within a Pod)
cat audit.log | jq 'select(
  (.user.username | startswith("system:serviceaccount:")) and
  .objectRef.subresource=="exec"
) | {
  sa: .user.username,
  targetPod: .objectRef.name,
  namespace: .objectRef.namespace
}'

Expected vs Unexpected SA Activity

ServiceAccount Expected Behavior Suspicious Behavior
kube-system:default Should NOT be making API calls (unused SA) Any API call at all
kube-system:coredns Watch services and endpoints Accessing secrets, creating pods
default:default Should NOT be making API calls Any API call (auto-mounted token)
app-namespace:app-sa Get specific configmaps/secrets Listing all secrets, exec into other pods

Security Best Practice: Disable automatic ServiceAccount token mounting for SAs that do not need API access:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: no-api-access
automountServiceAccountToken: false

4. Correlating Audit Logs with RBAC

Audit logs tell you what happened. RBAC tells you what was allowed. Correlating both reveals misconfigurations and overly permissive access.

Finding the RBAC Binding That Allowed an Action

The audit event’s annotations field contains the RBAC decision:

cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  .verb=="list"
) | {
  user: .user.username,
  decision: .annotations["authorization.k8s.io/decision"],
  reason: .annotations["authorization.k8s.io/reason"]
}'

Example output:

{
  "user": "system:serviceaccount:default:app-sa",
  "decision": "allow",
  "reason": "RBAC: allowed by RoleBinding \"app-secret-access\" of Role \"secret-reader\" to ServiceAccount \"app-sa\" in namespace \"default\""
}

Identifying Overly Permissive RBAC

# Find which RBAC bindings are authorizing sensitive actions
cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  .annotations["authorization.k8s.io/decision"]=="allow"
) | .annotations["authorization.k8s.io/reason"]' | sort | uniq -c | sort -rn

If you see a ClusterRoleBinding granting cluster-admin to a ServiceAccount that only needs to read configmaps, you have found a privilege escalation risk.

Detecting Denied Requests That Should Have Been Allowed

# 403 errors from known service accounts (potential RBAC misconfiguration)
cat audit.log | jq 'select(
  .responseStatus.code==403 and
  (.user.username | startswith("system:serviceaccount:"))
) | {
  sa: .user.username,
  verb: .verb,
  resource: .objectRef.resource,
  namespace: .objectRef.namespace,
  reason: .annotations["authorization.k8s.io/reason"]
}'

5. Alert Thresholds — Defining Suspicious Activity

Not every audit event is suspicious. You need to define thresholds that separate normal operations from potential threats.

Condition Threshold Action
Failed authentication (401) > 10 in 5 minutes from same IP Alert, investigate source IP
Forbidden responses (403) > 20 in 5 minutes from same user Alert, possible permission probing
Anonymous API access Any successful (2xx) anonymous request Alert immediately
Secret listing across namespaces Same user lists secrets in > 3 namespaces in 10 minutes Alert, potential credential harvesting
ClusterRoleBinding creation Any new ClusterRoleBinding with cluster-admin Alert immediately
Pod exec from unexpected SA ServiceAccount not in allow-list uses exec Alert, lateral movement
API access from external IP Source IP not in known CIDR ranges Alert, external access to API server

Implementing Threshold Checks

# Count 401 errors per source IP in the last 5 minutes
cat audit.log | jq 'select(
  .responseStatus.code==401 and
  .requestReceivedTimestamp > "2025-02-09T14:25:00Z"
) | .sourceIPs[0]' | sort | uniq -c | sort -rn

# Count secret access per user
cat audit.log | jq 'select(
  .objectRef.resource=="secrets"
) | .user.username' | sort | uniq -c | sort -rn

# Find users accessing secrets in multiple namespaces
cat audit.log | jq 'select(
  .objectRef.resource=="secrets" and
  (.verb | IN("get","list"))
) | {user: .user.username, ns: .objectRef.namespace}' | sort -u | jq -s 'group_by(.user) | map(select(length > 3)) | .[] | .[0].user'

6. Real-Time Monitoring Patterns

Tail the Audit Log

# Watch audit events in real-time
tail -f /var/log/kubernetes/audit/audit.log | jq '{
  user: .user.username,
  verb: .verb,
  resource: .objectRef.resource,
  name: .objectRef.name,
  code: .responseStatus.code
}'

# Watch only secret access in real-time
tail -f /var/log/kubernetes/audit/audit.log | jq 'select(.objectRef.resource=="secrets") | {
  user: .user.username,
  verb: .verb,
  name: .objectRef.name,
  namespace: .objectRef.namespace
}'

# Watch for 403 responses in real-time
tail -f /var/log/kubernetes/audit/audit.log | jq 'select(.responseStatus.code==403) | {
  user: .user.username,
  verb: .verb,
  resource: .objectRef.resource
}'

API Server Metrics for Audit

The API server exposes metrics that complement audit logs:

Metric Description
apiserver_audit_event_total Total number of audit events generated
apiserver_audit_error_total Total number of audit events that failed to be sent to the backend
apiserver_audit_requests_rejected_total Number of requests rejected due to audit policy errors
apiserver_request_total Total API requests (by verb, resource, code) 
# Check audit metrics (if metrics endpoint is accessible)
kubectl get --raw /metrics | grep apiserver_audit

7. Dashboard and Alerting Architecture

┌────────────────────────────────────────────────────┐
│                  kube-apiserver                      │
│   audit-log-path=/var/log/kubernetes/audit/audit.log│
└──────────┬─────────────────────────────────────────┘
           │
           ▼
┌──────────────────────┐    ┌────────────────────────┐
│  Fluent Bit / Fluentd │───▶│    Elasticsearch       │
│  (log collector)      │    │    (log storage)        │
└──────────────────────┘    └────────┬───────────────┘
                                     │
                            ┌────────▼───────────────┐
                            │    Kibana / Grafana      │
                            │    (visualization)       │
                            └────────┬───────────────┘
                                     │
                            ┌────────▼───────────────┐
                            │    Alert Manager         │
                            │    → Slack, PagerDuty    │
                            └────────────────────────┘

8. Practice Questions

Question 1

How do you identify anonymous API access in audit logs, and why is successful anonymous access a security concern?

Show Answer

Anonymous requests have user.username set to system:anonymous and user.groups containing system:unauthenticated.

To find them:

cat audit.log | jq 'select(.user.username == "system:anonymous")'

Successful anonymous access (HTTP 2xx response) is a security concern because it means:

  1. The API server is configured with --anonymous-auth=true (the default)
  2. RBAC has granted permissions to the system:unauthenticated group or the system:anonymous user
  3. An unauthenticated attacker can access cluster resources without any credentials

Even read-only access is dangerous — an anonymous user listing pods, services, or endpoints can map the entire cluster topology to plan further attacks.

Question 2

A ServiceAccount system:serviceaccount:default:app-sa is accessing secrets in the kube-system namespace. How do you investigate this with audit logs?

Show Answer 
# Find all secret access by this SA in kube-system
cat audit.log | jq 'select(
  .user.username=="system:serviceaccount:default:app-sa" and
  .objectRef.resource=="secrets" and
  .objectRef.namespace=="kube-system"
) | {
  time: .requestReceivedTimestamp,
  verb: .verb,
  secret: .objectRef.name,
  code: .responseStatus.code,
  reason: .annotations["authorization.k8s.io/reason"]
}'

Investigation steps:

  1. Check the RBAC reason — the authorization.k8s.io/reason annotation shows which RoleBinding/ClusterRoleBinding authorized the access. A SA in the default namespace should not have RoleBindings in kube-system.

  2. Check for ClusterRoleBindings — the SA may have a ClusterRoleBinding granting cluster-wide secret access: 
    kubectl get clusterrolebindings -o json | jq '.items[] | select(.subjects[]? | select(.name=="app-sa" and .namespace=="default"))'

  3. Assess the impact — kube-system secrets include ServiceAccount tokens for controllers, TLS certificates, and cloud provider credentials. Access to these secrets is a critical privilege escalation vector.

  4. Remediate — remove the overly permissive binding and restrict the SA to namespace-scoped Roles in default only.

Question 3

What API server metrics can you use to monitor audit event generation and detect problems with the audit backend?

Show Answer

Key metrics:

Metric Purpose
apiserver_audit_event_total Total audit events generated. A sudden drop might indicate audit logging is broken. A sudden spike indicates unusual API activity.
apiserver_audit_error_total Audit events that failed to be sent to the backend. Non-zero values mean you are losing audit events — a compliance and security problem.
apiserver_audit_requests_rejected_total Requests rejected because of audit policy errors. Indicates a misconfigured policy.

Monitor these with:

kubectl get --raw /metrics | grep apiserver_audit

In production, scrape these with Prometheus and set alerts:

  • apiserver_audit_error_total increasing → audit backend (file or webhook) is failing
  • apiserver_audit_event_total dropping to zero → audit logging may have stopped

Question 4

Describe how to correlate an audit log event with the specific RBAC binding that allowed the action.

Show Answer

Every allowed audit event includes an annotation authorization.k8s.io/reason that contains the RBAC decision details:

cat audit.log | jq '.annotations["authorization.k8s.io/reason"]'

Example output:

"RBAC: allowed by RoleBinding \"app-secret-access\" of Role \"secret-reader\" to ServiceAccount \"app-sa\" in namespace \"default\""

This tells you:

  • RoleBinding name: app-secret-access
  • Role name: secret-reader
  • Subject: ServiceAccount app-sa in namespace default

You can then inspect the Role to understand exactly what permissions were granted:

kubectl describe role secret-reader -n default
kubectl describe rolebinding app-secret-access -n default

This correlation is essential for answering: “Was this access intended, or is the RBAC binding overly permissive?”

Question 5

You need to set up monitoring to detect when any user runs kubectl exec into a pod in the production namespace. What audit events do you look for, and what alert would you configure?

Show Answer

Audit events for kubectl exec:

  • objectRef.resource = pods
  • objectRef.subresource = exec
  • verb = create
  • objectRef.namespace = production

Real-time monitoring:

tail -f /var/log/kubernetes/audit/audit.log | jq 'select(
  .objectRef.resource=="pods" and
  .objectRef.subresource=="exec" and
  .objectRef.namespace=="production"
) | {
  time: .requestReceivedTimestamp,
  user: .user.username,
  pod: .objectRef.name,
  sourceIP: .sourceIPs[0]
}'

Alert configuration:

  1. Forward audit logs to a centralized system (Elasticsearch, Splunk, or a webhook).
  2. Create an alert rule that triggers on any event matching the above criteria.
  3. Alert immediately for exec events in production — this is a critical environment where interactive shell access should be rare and controlled.
  4. Include context in the alert: user identity, target pod, source IP, and timestamp.
  5. Maintain an allow-list of users/SAs expected to use exec in production. Alert on any exec event from identities NOT on the allow-list.
Previous Up Next
Audit Log Analysis CKS Home Audit Best Practices

Updated:

You may also enjoy