CKS — Threat Detection
Threat Detection
Runtime threat detection goes beyond monitoring individual syscalls — it maps attacker behaviors to known attack patterns and triggers alerts when those patterns are observed. This section covers the MITRE ATT&CK framework for containers, Falco rules for each attack phase, behavioral analytics, incident response, and alert routing with Falcosidekick.
MITRE ATT&CK for Containers
The MITRE ATT&CK framework categorizes attacker tactics and techniques. Microsoft adapted it for Kubernetes environments. Understanding these phases helps you build detection rules for each stage of an attack.
Attack Kill Chain in Kubernetes:
┌──────────────┐
│ Initial │ Exposed dashboard, vulnerable image, compromised credentials
│ Access │
└──────┬───────┘
│
┌──────▼───────┐
│ Execution │ kubectl exec, reverse shell, malicious container
│ │
└──────┬───────┘
│
┌──────▼───────┐
│ Persistence │ Create new pod, backdoor image, modify kubelet
│ │
└──────┬───────┘
│
┌──────▼───────┐
│ Privilege │ Mount host filesystem, container escape, exploit CVE
│ Escalation │
└──────┬───────┘
│
┌──────▼───────┐
│ Lateral │ Service account token, network scanning,
│ Movement │ access other pods/nodes
└──────┬───────┘
│
┌──────▼───────┐
│ Exfiltration │ Data theft, DNS tunneling, external upload
│ │
└──────────────┘
Phase 1 — Initial Access
How attackers get in:
- Exposed Kubernetes dashboard without authentication
- Vulnerable application with RCE (Remote Code Execution)
- Compromised container image from public registry
- Stolen kubeconfig or service account token
- Misconfigured RBAC allowing anonymous access
Falco rule — Detect container started from untrusted image:
- list: trusted_registries
items: [docker.io/library, gcr.io/myproject, mycompany.azurecr.io]
- rule: Container from untrusted registry
desc: Detect container started from a registry not in the trusted list
condition: >
container and evt.type = container
and not (container.image.repository startswith docker.io/library
or container.image.repository startswith gcr.io/myproject
or container.image.repository startswith mycompany.azurecr.io)
output: >
Container from untrusted registry
(image=%container.image.repository container=%container.name
namespace=%k8s.ns.name pod=%k8s.pod.name)
priority: WARNING
tags: [container, mitre_initial_access]
Phase 2 — Execution
How attackers execute commands:
kubectl execinto a container- Reverse shell from an exploited application
- Malicious init container or sidecar
- Web shell uploaded through application vulnerability
Falco rule — Detect shell spawned in container:
- rule: Shell spawned in container
desc: Detect any shell process started in a container
condition: >
spawned_process and container
and proc.name in (bash, sh, zsh, dash, ash, csh, ksh)
and proc.tty != 0
output: >
Shell spawned in container (user=%user.name shell=%proc.name
parent=%proc.pname command=%proc.cmdline
container=%container.name image=%container.image.repository
namespace=%k8s.ns.name pod=%k8s.pod.name)
priority: WARNING
tags: [container, shell, mitre_execution]
Falco rule — Detect reverse shell patterns:
- rule: Reverse shell detected
desc: Detect reverse shell execution in a container
condition: >
spawned_process and container
and (
(proc.cmdline contains "/dev/tcp/" and proc.cmdline contains "bash")
or (proc.name = python and proc.cmdline contains "socket" and proc.cmdline contains "subprocess")
or (proc.name in (nc, ncat, netcat) and (proc.args contains "-e" or proc.args contains "-c"))
or (proc.name = perl and proc.cmdline contains "socket" and proc.cmdline contains "exec")
)
output: >
REVERSE SHELL detected (user=%user.name command=%proc.cmdline
container=%container.name namespace=%k8s.ns.name pod=%k8s.pod.name)
priority: CRITICAL
tags: [container, network, mitre_execution]
Phase 3 — Persistence
How attackers maintain access:
- Create a new privileged pod with backdoor
- Push modified image to registry
- Add cron job inside a container
- Modify kubelet configuration
- Create a ClusterRoleBinding for their service account
Falco rule — Detect writes to cron directories:
- rule: Cron job created in container
desc: Detect creation or modification of cron files in a container
condition: >
open_write and container
and (fd.name startswith /etc/cron or fd.name startswith /var/spool/cron)
output: >
Cron job modified in container (user=%user.name command=%proc.cmdline
file=%fd.name container=%container.name namespace=%k8s.ns.name)
priority: CRITICAL
tags: [container, filesystem, mitre_persistence]
Falco rule — Detect new privileged pod creation via API:
- rule: Privileged pod created
desc: Detect creation of privileged pods through the Kubernetes API
condition: >
kevt and kcreate and pod
and ka.req.pod.containers.privileged = true
output: >
Privileged pod created (user=%ka.user.name pod=%ka.target.name
namespace=%ka.target.namespace image=%ka.req.pod.containers.image)
priority: CRITICAL
tags: [k8s, mitre_persistence]
Phase 4 — Privilege Escalation
How attackers escalate privileges:
- Mount host filesystem (
hostPath: /) - Container escape via kernel exploit
- Use
nsenterto access host namespace - Exploit setuid binaries
- Use service account with cluster-admin
Falco rule — Detect host filesystem mount access:
- rule: Access to host mount
desc: Detect access to sensitive host paths mounted in a container
condition: >
open_read and container
and (fd.name startswith /host/ or fd.name startswith /hostroot/)
output: >
Host mount accessed (user=%user.name command=%proc.cmdline
file=%fd.name container=%container.name namespace=%k8s.ns.name)
priority: CRITICAL
tags: [container, filesystem, mitre_privilege_escalation]
Falco rule — Detect nsenter usage:
- rule: Nsenter detected
desc: Detect nsenter command which can be used for container escape
condition: >
spawned_process and proc.name = nsenter
output: >
nsenter executed (user=%user.name command=%proc.cmdline
container=%container.name namespace=%k8s.ns.name pod=%k8s.pod.name)
priority: CRITICAL
tags: [container, process, mitre_privilege_escalation]
Phase 5 — Lateral Movement
How attackers move laterally:
- Read service account token and call Kubernetes API
- Scan cluster network for other services
- Access other pods via cluster-internal DNS
- Exploit trusting services (e.g., databases without auth)
- Use stolen secrets to access external systems
Falco rule — Detect service account token access:
- rule: Service account token accessed
desc: Detect process reading the Kubernetes service account token
condition: >
open_read and container
and fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount
and not proc.name in (pause)
output: >
Service account token read (user=%user.name command=%proc.cmdline
file=%fd.name container=%container.name namespace=%k8s.ns.name
pod=%k8s.pod.name)
priority: WARNING
tags: [container, k8s, mitre_credential_access]
Falco rule — Detect network scanning tools:
- list: network_scanners
items: [nmap, masscan, zmap, netcat, nc, ncat]
- rule: Network scanning tool in container
desc: Detect network scanning tools executed in a container
condition: >
spawned_process and container
and proc.name in (network_scanners)
output: >
Network scanning tool detected (user=%user.name command=%proc.cmdline
container=%container.name namespace=%k8s.ns.name pod=%k8s.pod.name)
priority: CRITICAL
tags: [container, network, mitre_discovery]
Phase 6 — Exfiltration
How attackers steal data:
- Copy data to external server via
curl/wget - DNS tunneling (encode data in DNS queries)
- Upload to cloud storage
- Exfiltrate via DNS
TXTrecords - Reverse shell with data piped out
Falco rule — Detect data transfer tools:
- list: data_transfer_tools
items: [curl, wget, ftp, sftp, scp, rsync, rclone]
- rule: Data transfer tool in container
desc: Detect data exfiltration tools in a container
condition: >
spawned_process and container
and proc.name in (data_transfer_tools)
and not k8s.ns.name in (kube-system)
output: >
Data transfer tool executed (user=%user.name command=%proc.cmdline
container=%container.name namespace=%k8s.ns.name pod=%k8s.pod.name)
priority: WARNING
tags: [container, network, mitre_exfiltration]
Falco rule — Detect unusual outbound connections:
- rule: Unexpected external connection
desc: Detect outbound connections to non-cluster IPs
condition: >
evt.type = connect and container
and fd.typechar = 4
and not fd.sip in (rfc1918_addresses)
and not k8s.ns.name in (kube-system, monitoring)
output: >
External connection from container (user=%user.name command=%proc.cmdline
dest=%fd.name container=%container.name namespace=%k8s.ns.name
pod=%k8s.pod.name)
priority: WARNING
tags: [container, network, mitre_exfiltration]
Behavioral Analytics — Baseline and Detect
Building effective threat detection requires understanding normal behavior so you can detect anomalous behavior.
Baseline → Detect workflow:
| Step | Action | Tool |
|---|---|---|
| 1 | Profile normal processes in each container | sysdig -c spy_users container.name=X |
| 2 | Identify normal network connections | sysdig "evt.type=connect and container.name=X" |
| 3 | Record normal file access patterns | sysdig "evt.type in (open,openat) and container.name=X" |
| 4 | Create Falco rules that detect deviations | Custom rules with allowlists |
| 5 | Tune rules to reduce false positives | Exclude known-good processes/paths |
Example — Baseline-based rule:
# This app should ONLY run: node, npm, and puma processes
- list: allowed_app_processes
items: [node, npm, puma]
- rule: Unexpected process in web-app
desc: Detect processes not in the expected baseline
condition: >
spawned_process and container
and container.image.repository = "mycompany/web-app"
and not proc.name in (allowed_app_processes)
output: >
Unexpected process (proc=%proc.name command=%proc.cmdline
container=%container.name namespace=%k8s.ns.name)
priority: WARNING
tags: [container, process, behavioral]
Incident Response Workflow
When Falco fires an alert, follow a structured incident response process:
┌──────────┐ ┌──────────┐ ┌─────────────┐ ┌──────────┐
│ Detect │───▶│ Contain │───▶│ Investigate │───▶│ Remediate│
└──────────┘ └──────────┘ └─────────────┘ └──────────┘
Step 1 — Detect
Falco alert fires. Identify the affected pod, namespace, and node:
journalctl -u falco --since "5 minutes ago" | grep CRITICAL
Step 2 — Contain
Immediately limit the blast radius:
# Delete the compromised pod
kubectl delete pod <pod-name> -n <namespace>
# OR isolate via network policy (deny all traffic)
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: isolate-compromised
namespace: <namespace>
spec:
podSelector:
matchLabels:
app: <compromised-app>
policyTypes:
- Ingress
- Egress
EOF
# Cordon the node to prevent new pods
kubectl cordon <node-name>
Step 3 — Investigate
Gather forensic data:
# Check Falco logs for the full timeline
journalctl -u falco --since "1 hour ago" | grep <pod-name>
# Capture syscalls with sysdig if the pod is still running
sudo sysdig -w /tmp/forensics.scap container.name=<container-name>
# Check Kubernetes audit logs
kubectl logs -n kube-system kube-apiserver-<node> | grep <pod-name>
# Review RBAC — what could the service account do?
kubectl auth can-i --list --as system:serviceaccount:<ns>:<sa>
Step 4 — Remediate
Fix the root cause:
# Update RBAC to remove excessive permissions
# Patch the vulnerability in the container image
# Add network policies to restrict communication
# Add Falco rules to detect the specific attack pattern
# Enable immutability settings on the workload
Falcosidekick — Alert Routing
Falcosidekick is a companion tool that receives Falco alerts and forwards them to 50+ destinations. This is how you integrate Falco with your organization’s alerting pipeline.
Architecture:
Falco → HTTP Output → Falcosidekick → Slack
→ PagerDuty
→ Elasticsearch
→ AWS CloudWatch
→ Prometheus/Alertmanager
→ Webhook
→ OpsGenie
→ Microsoft Teams
Configure Falco to send alerts to Falcosidekick:
Edit /etc/falco/falco.yaml:
http_output:
enabled: true
url: http://localhost:2801 # Falcosidekick default port
Install Falcosidekick via Helm:
helm install falcosidekick falcosecurity/falcosidekick \
--namespace falco \
--set config.slack.webhookurl="https://hooks.slack.com/services/XXX" \
--set config.slack.minimumpriority="warning"
Falcosidekick configuration for Slack:
slack:
webhookurl: "https://hooks.slack.com/services/T00/B00/XXX"
minimumpriority: "warning"
channel: "#security-alerts"
username: "Falco"
icon: "https://falco.org/favicon.png"
Falcosidekick configuration for multiple outputs:
slack:
webhookurl: "https://hooks.slack.com/services/..."
minimumpriority: "warning"
pagerduty:
routingkey: "your-pagerduty-key"
minimumpriority: "critical"
elasticsearch:
hostport: "http://elasticsearch:9200"
index: "falco-alerts"
minimumpriority: "notice"
webhook:
address: "https://my-siem.example.com/api/alerts"
minimumpriority: "warning"
Complete Detection Rule Set — Production Template
A comprehensive set of rules for a production cluster:
# /etc/falco/falco_rules.local.yaml
# === Lists ===
- list: trusted_registries
items: [docker.io/library, gcr.io/myproject, mycompany.azurecr.io]
- list: package_managers
items: [apt, apt-get, yum, dnf, apk, pip, pip3, npm]
- list: network_tools
items: [nmap, masscan, nc, ncat, netcat, socat, curl, wget]
- list: crypto_miners
items: [xmrig, minerd, cpuminer, minergate]
# === Override default rules ===
- rule: Terminal shell in container
override:
priority: replace
priority: CRITICAL
# === Custom detection rules ===
- rule: Package manager in container
desc: Detect package installation in running container
condition: spawned_process and container and proc.name in (package_managers)
output: "Package manager (user=%user.name cmd=%proc.cmdline container=%container.name ns=%k8s.ns.name)"
priority: ERROR
tags: [container, mitre_execution]
- rule: Network tool in container
desc: Detect network reconnaissance tools
condition: spawned_process and container and proc.name in (network_tools)
output: "Network tool (user=%user.name cmd=%proc.cmdline container=%container.name ns=%k8s.ns.name)"
priority: WARNING
tags: [container, network, mitre_discovery]
- rule: Crypto miner detected
desc: Detect cryptocurrency mining processes
condition: spawned_process and container and proc.name in (crypto_miners)
output: "CRYPTO MINER (user=%user.name cmd=%proc.cmdline container=%container.name ns=%k8s.ns.name)"
priority: CRITICAL
tags: [container, crypto, mitre_execution]
- rule: Write to binary directory
desc: Detect writes to system binary directories
condition: >
open_write and container
and (fd.name startswith /bin or fd.name startswith /sbin
or fd.name startswith /usr/bin or fd.name startswith /usr/sbin)
output: "Binary dir write (user=%user.name cmd=%proc.cmdline file=%fd.name container=%container.name)"
priority: CRITICAL
tags: [container, filesystem, mitre_persistence]
Practice Questions
Question 1
Name the six phases of the MITRE ATT&CK framework as applied to Kubernetes attacks.
Show Answer
- Initial Access — gaining entry into the cluster (exposed dashboard, compromised image, stolen credentials)
- Execution — running commands in the cluster (kubectl exec, reverse shell, malicious container)
- Persistence — maintaining access (backdoor pod, modified image, cron job)
- Privilege Escalation — gaining higher privileges (host mount, container escape, setuid exploit)
- Lateral Movement — accessing other resources (service account token, network scanning, pod-to-pod)
- Exfiltration — stealing data (curl to external, DNS tunneling, cloud storage upload)
Question 2
Write a Falco rule to detect when nmap or masscan is executed inside any container.
Show Answer
- rule: Network scanner in container
desc: Detect network scanning tools in a container
condition: >
spawned_process and container
and proc.name in (nmap, masscan)
output: >
Network scanner detected (user=%user.name command=%proc.cmdline
container=%container.name namespace=%k8s.ns.name pod=%k8s.pod.name)
priority: CRITICAL
tags: [container, network, mitre_discovery]
Add to /etc/falco/falco_rules.local.yaml and restart:
sudo systemctl restart falco
Question 3
A Falco alert shows “Shell spawned in container” for pod web-app in namespace production. Describe the incident response steps.
Show Answer
Step 1 — Detect: Review the full Falco alert:
journalctl -u falco --since "10 minutes ago" | grep web-app
Step 2 — Contain: Isolate the pod immediately:
# Option A: Delete the pod
kubectl delete pod web-app -n production
# Option B: Network-isolate if investigation needed
kubectl apply -f deny-all-netpol.yaml
Step 3 — Investigate:
# Check what commands were run
journalctl -u falco | grep web-app
# Review RBAC for the service account
kubectl auth can-i --list --as system:serviceaccount:production:default
# Check if data was exfiltrated (network connections)
journalctl -u falco | grep "connect" | grep web-app
Step 4 — Remediate:
- Patch the vulnerability that allowed initial access
- Add
readOnlyRootFilesystem: trueto prevent future writes - Tighten RBAC and network policies
- Add targeted Falco rules for this specific workload
Question 4
What is Falcosidekick and how do you configure Falco to send alerts to it?
Show Answer
Falcosidekick is a tool that receives Falco alerts via HTTP and forwards them to 50+ output destinations (Slack, PagerDuty, Elasticsearch, webhooks, etc.).
Configure Falco in /etc/falco/falco.yaml:
http_output:
enabled: true
url: http://falcosidekick:2801
Install Falcosidekick via Helm:
helm install falcosidekick falcosecurity/falcosidekick \
--namespace falco \
--set config.slack.webhookurl="https://hooks.slack.com/services/XXX"
Restart Falco after changing the configuration:
sudo systemctl restart falco
Question 5
Why is behavioral baselining important for effective threat detection? Give an example.
Show Answer
Why: Generic rules produce too many false positives. By understanding what a container normally does, you can detect deviations that indicate compromise.
Example: A web server container normally runs only nginx and node processes. If bash, curl, or python suddenly appears, that is anomalous.
- list: expected_web_processes
items: [nginx, node]
- rule: Unexpected process in web container
desc: Detect processes outside the known baseline
condition: >
spawned_process and container
and container.image.repository = "mycompany/web-app"
and not proc.name in (expected_web_processes)
output: >
Unexpected process (proc=%proc.name cmd=%proc.cmdline
container=%container.name ns=%k8s.ns.name)
priority: WARNING
This is far more precise than a generic “detect all shells” rule — it catches any unexpected process, not just shells.
Next: Runtime Security Lab →
