KCSA — Security Observability Stack

7 minute read

Security Observability Stack

You cannot secure what you cannot see. Detection, investigation, and response all depend on observability — the ability to understand what is happening inside your cluster at any point in time. The KCSA exam treats observability as a core security capability, not just an operations concern.

Three Pillars of Observability

Observability rests on three complementary data types:

Pillar What It Captures Example Tools Security Application
Metrics Numerical measurements over time Prometheus, Grafana Alert on anomalies — CPU spikes, unusual request rates, failed auth attempts
Logs Discrete event records Elasticsearch, Fluentd, Kibana (EFK) Audit trails — who did what, when, and from where
Traces Request flow across services Jaeger, Zipkin, OpenTelemetry Detect unusual call patterns — unexpected service dependencies, lateral movement

Each pillar answers a different question:

  • Metrics → “Is something abnormal happening right now?”
  • Logs → “What exactly happened and who did it?”
  • Traces → “How did a request flow through the system?”

All three pillars are needed for complete security observability. Metrics alone miss context, logs alone miss patterns, and traces alone miss volume.

Security Monitoring Components

The cloud native ecosystem has specific tools for each observability layer, plus specialized security tools:

Component Tool Security Use Case
Metrics Prometheus + Grafana Alert on resource abuse, excessive failed logins, anomalous request rates
Logs EFK (Elasticsearch, Fluentd, Kibana) or ELK (Elasticsearch, Logstash, Kibana) Kubernetes audit logs, pod logs, security event aggregation
Traces Jaeger, Zipkin, OpenTelemetry Detect unusual service call patterns, identify lateral movement
Runtime Falco Syscall-level threat detection (covered in Falco chapter)
Network Cilium Hubble Network flow visibility — who talked to whom, on which ports

Prometheus for Security

Prometheus is the de facto standard for metrics collection in Kubernetes (CNCF Graduated). While primarily used for operational monitoring, it is equally valuable for security:

Security-Relevant Metrics

Metric Source Security Signal
apiserver_audit_event_total kube-apiserver Volume of audit events — spikes may indicate attack
apiserver_authentication_attempts kube-apiserver Failed authentication attempts — brute force detection
rest_client_requests_total components Unusual API request patterns
container_cpu_usage_seconds_total cAdvisor Cryptomining indicators (sustained high CPU)
container_network_transmit_bytes_total cAdvisor Data exfiltration indicators (unusual outbound traffic)
kube_pod_status_phase kube-state-metrics Pods in unexpected states (CrashLoopBackOff)

Prometheus Alerting Rules for Security

groups:
  - name: security-alerts
    rules:
      - alert: HighFailedAuthAttempts
        expr: rate(apiserver_authentication_attempts{result="failure"}[5m]) > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High rate of failed authentication attempts"
          description: "More than 10 failed auth attempts per second for 2 minutes"

      - alert: SuspiciousCPUUsage
        expr: |
          container_cpu_usage_seconds_total{namespace!="kube-system"}
          > 0.9 * kube_pod_container_resource_limits{resource="cpu"}
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Container consistently at CPU limit — potential cryptomining"

Prometheus metrics are numerical and lightweight. They tell you “something abnormal is happening” — but you need logs and traces to understand what and why.

Kubernetes Audit Logs

Kubernetes audit logs record every request to the API server. They are the single most important log source for Kubernetes security.

Audit Log Pipeline

User/Controller → kube-apiserver → Audit Policy → Audit Backend → Storage
                                       ↓
                              (what to log, at what level)
                                       ↓
                              ┌──────────────────┐
                              │   Log backend     │ → file on disk
                              │   Webhook backend │ → external system (EFK, SIEM)
                              └──────────────────┘

Audit Levels

Level What Is Recorded
None Don’t log the event
Metadata Log request metadata (user, timestamp, resource, verb) — no body
Request Log metadata + request body
RequestResponse Log metadata + request body + response body

Security-Relevant Audit Events

Event Why It Matters
Secret access (get, list, watch on secrets) Credential theft detection
RBAC changes (create, update on roles/bindings) Privilege escalation
Pod exec (create on pods/exec) Interactive shell access
Node modifications Infrastructure tampering
ServiceAccount token requests Lateral movement indicators 
# Example audit policy (conceptual)
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log all secret access at Request level
  - level: Request
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["get", "list", "watch"]

  # Log pod exec at RequestResponse level
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec"]

  # Log RBAC changes at Request level
  - level: Request
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
    verbs: ["create", "update", "patch", "delete"]

  # Log everything else at Metadata level
  - level: Metadata

Audit logs at the RequestResponse level capture the most detail — but generate the most volume. Balance detail with storage cost.

EFK/ELK Stack for Log Aggregation

Kubernetes generates logs across hundreds or thousands of pods. A centralized log aggregation system is required for security investigation:

EFK Stack

Component Role
Elasticsearch Search and storage engine — indexes and stores log data
Fluentd (or Fluent Bit) Log collector — runs as DaemonSet, ships logs from every node
Kibana Visualization — dashboards, search interface, alert management 
Node logs + Pod logs + Audit logs
         │
         ▼
    Fluentd (DaemonSet on every node)
         │
         ▼
    Elasticsearch (central storage)
         │
         ▼
    Kibana (search, dashboards, alerts)

Security Dashboards in Kibana

Dashboard What It Shows
Audit Log Overview API server request volume, top users, error codes
Failed Authentication Failed auth attempts over time, by user and source IP
Secret Access Who accessed which secrets, when
Pod Exec Activity Interactive shell sessions in containers
RBAC Changes Role and binding modifications over time

SIEM Integration

SIEM (Security Information and Event Management) platforms provide centralized security monitoring, correlation, and incident management. Common SIEMs include Splunk, Microsoft Sentinel, Elastic Security, and Sumo Logic.

Kubernetes → SIEM Pipeline

Kubernetes Cluster
    ├── Audit logs ──────→ Fluentd ──────→ SIEM
    ├── Falco alerts ────→ Falcosidekick ─→ SIEM
    ├── Pod logs ────────→ Fluentd ──────→ SIEM
    ├── Prometheus alerts → Alertmanager ─→ SIEM
    └── Network flows ───→ Hubble ───────→ SIEM

What SIEM Adds Beyond Raw Logging

Capability Description
Correlation Connect events across sources (audit log + Falco alert + network flow)
Enrichment Add context — user identity, pod ownership, namespace metadata
Detection rules Pre-built and custom rules for known attack patterns
Incident management Track investigation, assign responders, document findings
Compliance reporting Generate audit reports for standards like SOC 2, PCI DSS
Long-term retention Store events for months or years as required by regulations

For the KCSA exam, know that SIEM is the centralized brain of security operations — it aggregates, correlates, and alerts on security events from across the entire infrastructure.

Grafana Security Dashboards

Grafana is the standard visualization layer for Prometheus metrics. Security teams use Grafana dashboards to:

  • Monitor API server request rates and error codes
  • Track resource consumption anomalies (potential cryptomining)
  • Visualize network traffic patterns (data exfiltration indicators)
  • Display Falco alert trends over time
  • Show compliance posture metrics (policy violations from OPA/Kyverno)

Example Security Dashboard Panels

Panel Data Source Visualization
Failed Auth Rate Prometheus (apiserver metrics) Time-series graph
Active Falco Alerts Falco metrics via Prometheus Bar chart by severity
Policy Violations Kyverno/Gatekeeper metrics Table with violation count per policy
Network Traffic Cilium Hubble metrics Flow diagram
Pod Security Posture kube-state-metrics Gauge — % compliant pods

Alert Fatigue and Meaningful Alerting

One of the biggest challenges in security observability is alert fatigue — when too many alerts are generated, teams stop paying attention and real threats are missed.

Preventing Alert Fatigue

Strategy Description
Severity-based routing Only page on-call for CRITICAL/HIGH — send LOW/MEDIUM to dashboards
Alert aggregation Group related alerts (e.g., 100 Falco alerts from one compromised pod = one incident)
Tuning thresholds Adjust alert thresholds based on baseline behavior — not arbitrary numbers
Suppressing known noise Silence expected patterns (scheduled jobs, maintenance windows)
Runbooks Attach actionable investigation steps to every alert

The goal is not zero alerts — it is meaningful alerts that drive action.

Incident Detection → Investigation → Response

Security observability enables a structured incident workflow:

Detection                Investigation              Response
─────────────          ──────────────────          ──────────────
Prometheus alert   →   Query audit logs        →   Isolate compromised pod
Falco alert        →   Check Falco output      →   Revoke credentials
Kibana anomaly     →   Trace request flow      →   Patch vulnerability
SIEM correlation   →   Identify blast radius   →   Conduct post-mortem

The OODA Loop for Security Incidents

Phase Action Tools
Observe Detect anomaly via alert or dashboard Prometheus, Falco, SIEM
Orient Investigate scope and impact Kibana, Jaeger, audit logs
Decide Determine response action Incident runbook, security team
Act Execute response (isolate, remediate, report) kubectl, policy update, SIEM ticket

Network Observability with Cilium Hubble

Hubble is the observability layer of Cilium (CNCF Graduated CNI plugin). It provides deep visibility into network flows:

Capability Description
Flow logs Every network connection — source, destination, port, protocol, verdict
DNS visibility DNS queries and responses from pods
HTTP/gRPC visibility L7 request/response details
Flow visualization Hubble UI shows real-time service communication maps
Policy verdict Shows which NetworkPolicy allowed or denied each flow 
# View flows with Hubble CLI (conceptual)
hubble observe --namespace production --verdict DROPPED
hubble observe --to-pod backend --protocol TCP --port 443

Hubble shows network flows with policy context — you can see not just that traffic was dropped, but which NetworkPolicy dropped it and why.

Key Takeaways for the Exam

  1. Three pillars of observability — metrics (Prometheus), logs (EFK/ELK), traces (Jaeger)
  2. Kubernetes audit logs are critical — they record every API server request at configurable detail levels
  3. Audit levels — None, Metadata, Request, RequestResponse (increasing detail and volume)
  4. Prometheus alerts can detect security anomalies — failed auth, CPU abuse, unusual traffic
  5. EFK stack — Fluentd collects, Elasticsearch stores, Kibana visualizes (DaemonSet deployment)
  6. SIEM provides correlation — connects events across multiple sources into a unified security view
  7. Alert fatigue is a real risk — severity routing, aggregation, and tuning are essential
  8. Grafana dashboards visualize security posture using Prometheus, Falco, and policy metrics
  9. Hubble (Cilium) provides network flow visibility with policy verdict context
  10. Incident workflow — Detect → Investigate → Respond → Post-mortem

Next: Practice Questions — Cloud Native Security Tools

Updated:

You may also enjoy