CKS — Syscall Monitoring

9 minute read

Syscall Monitoring

Every action a container takes — reading a file, opening a network connection, spawning a process — is a system call (syscall). The Linux kernel provides hundreds of syscalls, and monitoring them is the foundation of runtime security. Falco, sysdig, and strace all operate at this layer.

For the CKS exam, you need to understand which syscalls indicate which threats, how Falco intercepts them, and how to use complementary tools to investigate suspicious activity.

How Syscalls Work

When a process inside a container needs to interact with the operating system (read a file, create a socket, fork a child process), it makes a syscall. The kernel handles the request and returns a result.

┌──────────────────────────────────┐
│  Container Process               │
│  cat /etc/shadow                 │
│     ↓                            │
│  libc: open("/etc/shadow", ...) │
└──────────┬───────────────────────┘
           │  syscall: openat()
┌──────────▼───────────────────────┐
│  Linux Kernel                     │
│  ┌───────────────────────────┐   │
│  │ Falco Driver (kmod/eBPF)  │   │
│  │ Captures event + metadata │   │
│  └───────────┬───────────────┘   │
│              │                    │
│  Kernel processes the syscall    │
└──────────────────────────────────┘
           │
           ▼
  Falco Engine evaluates rules

Key insight: Containers share the host kernel. There is no separate kernel per container. This means a single Falco instance on the host can monitor all containers running on that node.

How Falco Intercepts Syscalls

Falco uses one of two drivers to intercept syscalls before the kernel processes them:

Kernel Module (kmod)

Application → Syscall → Falco Kernel Module → Kernel → Result
                              ↓
                         Event Buffer → Falco User Space
  • Loaded via insmod or falco-driver-loader module
  • Fast, direct access to syscall table
  • Risk: a buggy module can crash the kernel
  • Requires kernel headers for compilation

eBPF Probe

Application → Syscall → eBPF Tracepoint → Kernel → Result
                              ↓
                         eBPF Map → Falco User Space
  • Loaded via the BPF subsystem
  • Verified by the kernel verifier — cannot crash the kernel
  • Requires kernel ≥ 4.14 with BPF support
  • Slightly higher overhead than kmod

Common Syscalls and Their Security Implications

Syscall What It Does Security Relevance
execve Execute a program Process spawning — shell access, malware execution
open / openat Open a file File access — reading credentials, writing malware
read Read from file descriptor Data exfiltration, credential theft
write Write to file descriptor File modification, log tampering
connect Initiate network connection C2 beaconing, data exfiltration, reverse shell
accept Accept incoming connection Backdoor listener
bind Bind to a port Unauthorized service
clone / fork Create child process Fork bomb, spawning malicious processes
setuid / setgid Change user/group ID Privilege escalation
mount Mount filesystem Container escape via host filesystem
ptrace Trace/debug process Process injection, debugging to extract secrets
unlink Delete a file Evidence destruction
rename Rename a file Covering tracks
socket Create network socket Network activity setup
sendto / recvfrom Send/receive data Data transfer — exfiltration or C2 communication

Syscall-to-Threat Mapping

Attack Syscalls Involved Falco Detection
Shell access execve (bash/sh) spawned_process and container and shell_procs
Credential theft openat, read (/etc/shadow) open_read and sensitive_files
Reverse shell execve, connect, dup2 spawned_process and proc.cmdline contains "/dev/tcp"
Container escape mount, open (/proc/1/root) evt.type = mount and container
Crypto mining execve (xmrig), connect (pool) proc.name in (xmrig, minerd)
Privilege escalation setuid, setgid evt.type = setuid and container
Data exfiltration connect, sendto evt.type = connect and fd.sip.name = <external>
Persistence open_write (/etc/crontab) open_write and fd.name startswith /etc/cron
Service account abuse openat, read (token path) fd.name startswith /var/run/secrets/kubernetes.io
Log tampering unlink, write (/var/log) (evt.type = unlink or open_write) and fd.name startswith /var/log

Creating Detection Rules for Specific Syscall Patterns

Detect Mount Syscall in Containers (Container Escape Attempt)

- rule: Mount attempted in container
  desc: Detect mount syscall inside a container (potential escape)
  condition: >
    evt.type = mount and container
  output: >
    Mount attempted in container (user=%user.name command=%proc.cmdline
     container=%container.name namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: CRITICAL
  tags: [container, filesystem, mitre_privilege_escalation]

Detect ptrace in Containers (Process Injection)

- rule: Ptrace detected in container
  desc: Detect ptrace syscall in a container (debugging/injection)
  condition: >
    evt.type = ptrace and container
    and not proc.name in (strace, gdb, ltrace)
  output: >
    Ptrace detected (user=%user.name command=%proc.cmdline
     container=%container.name namespace=%k8s.ns.name)
  priority: CRITICAL
  tags: [container, process, mitre_privilege_escalation]

Detect Outbound Connections from Restricted Pods

- rule: Unexpected outbound connection
  desc: Detect outbound network connections from containers not expected to make them
  condition: >
    evt.type = connect and container
    and fd.typechar = 4
    and not fd.sip.name in (kubernetes.default.svc.cluster.local)
    and k8s.ns.name = "restricted"
  output: >
    Outbound connection from restricted namespace
    (user=%user.name command=%proc.cmdline dest=%fd.name
     container=%container.name namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, network, mitre_exfiltration]

Sysdig — Complementary Syscall Tool

Sysdig is an open-source tool from the same company that created Falco. While Falco is a real-time rule-based alerting engine, sysdig is a capture and inspection tool — think of it as tcpdump for syscalls.

Installing sysdig:

sudo apt-get install -y sysdig

Common sysdig commands for investigation:

# Watch all syscalls from a specific container
sudo sysdig container.name=nginx

# Watch only execve syscalls (process launches) in containers
sudo sysdig "evt.type=execve and container"

# Spy on users — see every command executed
sudo sysdig -c spy_users

# Watch file opens in a specific container
sudo sysdig "evt.type in (open, openat) and container.name=web-app"

# Watch network activity in containers
sudo sysdig "evt.type in (connect, accept) and container"

# Capture syscalls to a file for later analysis
sudo sysdig -w /tmp/capture.scap "container.name=suspicious-pod"

# Read captured syscalls
sudo sysdig -r /tmp/capture.scap

# Filter captured data
sudo sysdig -r /tmp/capture.scap "evt.type=execve"

Sysdig chisels (built-in scripts):

# List all available chisels
sudo sysdig -cl

# Top processes by CPU
sudo sysdig -c topprocs_cpu

# Top files by read bytes
sudo sysdig -c topfiles_bytes

# See all commands users execute
sudo sysdig -c spy_users

# Network connections per container
sudo sysdig -c topconns

Using strace for Targeted Debugging

strace traces syscalls for a single process. Useful for debugging a specific container process:

# Trace all syscalls for a process
strace -p <PID>

# Trace only file-related syscalls
strace -e trace=file -p <PID>

# Trace only network-related syscalls
strace -e trace=network -p <PID>

# Trace with timestamps
strace -t -p <PID>

# Trace a command directly
strace ls /etc/

# Trace syscalls inside a container (find PID first)
# From the host:
PID=$(docker inspect --format '{{.State.Pid}}' <container_id>)
sudo strace -p $PID -f

strace vs sysdig vs Falco:

Tool Purpose Scope Real-time Alerts
strace Debug a single process One PID No
sysdig Capture and inspect syscalls All containers on host No (manual inspection)
Falco Rule-based runtime alerting All containers on host Yes

Performance Impact of Syscall Monitoring

Syscall monitoring adds overhead. Understanding the tradeoffs:

Factor Kernel Module eBPF Probe No Monitoring
CPU overhead 1-3% 2-5% 0%
Latency per syscall ~1-2μs ~2-4μs 0
Memory usage Moderate Moderate None
Safety Can crash kernel Cannot crash kernel N/A
Hot reload No — restart needed Possible N/A

Best practices to minimize overhead:

  1. Use specific conditions in rules — avoid overly broad rules
  2. Limit the number of enabled rules to what you need
  3. Use macros and lists to optimize condition evaluation
  4. Choose eBPF for safety-critical environments
  5. Monitor Falco’s own resource usage: systemctl status falco

Filtering Syscalls by Container and Namespace

Falco conditions can target specific containers or namespaces:

# Only monitor a specific namespace
- rule: Shell in production
  condition: >
    spawned_process and container and shell_procs
    and k8s.ns.name = "production"

# Exclude system namespaces
- rule: Shell in user namespace
  condition: >
    spawned_process and container and shell_procs
    and not k8s.ns.name in (kube-system, kube-public, kube-node-lease)

# Monitor a specific container image
- rule: Shell in custom app
  condition: >
    spawned_process and container and shell_procs
    and container.image.repository = "mycompany/secure-app"

# Monitor by pod label (if Falco K8s metadata enrichment is enabled)
- rule: Shell in labeled pod
  condition: >
    spawned_process and container and shell_procs
    and k8s.pod.label.security = "high"

Examining Falco’s Syscall Event Stream

You can run Falco in a mode that prints raw events (useful for understanding what data is available):

# Print all events Falco sees (very verbose — use briefly)
sudo falco -A

# Print events matching a specific rule
sudo falco -A -t shell

# Print events in JSON format for machine parsing
sudo falco -A -o json_output=true

Practical Workflow: Investigating a Suspicious Container

  1. Falco alert fires — “Shell spawned in container”:
Warning A shell was spawned in container
  (container=web-app namespace=production pod=web-app-7d4f)
  1. Check what’s happening in the container with sysdig:
sudo sysdig container.name=web-app "evt.type=execve"
  1. See what files are being accessed:
sudo sysdig container.name=web-app "evt.type in (open, openat)"
  1. Check for network connections:
sudo sysdig container.name=web-app "evt.type in (connect, sendto)"
  1. Capture traffic for forensic analysis:
sudo sysdig -w /tmp/incident-capture.scap container.name=web-app
  1. Take action: Delete the pod, cordon the node, review logs.

Practice Questions

Question 1

Which syscall is triggered when someone executes kubectl exec -it pod -- /bin/bash?

Show Answer

The primary syscall is execve — it executes the /bin/bash program inside the container. Specifically:

  1. clone or fork creates a new process
  2. execve replaces the process image with /bin/bash
  3. read/write on the terminal PTY for I/O

Falco detects this via the spawned_process macro which checks evt.type = execve and evt.dir = <.

Question 2

An attacker inside a container runs cat /etc/shadow. Which syscalls are involved and which Falco rule detects this?

Show Answer

Syscalls involved:

  • execve — to start the cat process
  • openat — to open /etc/shadow for reading
  • read — to read the file contents
  • write — to write the output to stdout

Falco detection: The default rule “Read sensitive file untouched” fires because:

  • open_read matches the openat call
  • sensitive_files macro matches /etc/shadow
  • container macro confirms it’s inside a container

Question 3

What is the difference between sysdig and Falco? When would you use each?

Show Answer
Aspect Falco sysdig
Purpose Real-time rule-based alerting Syscall capture and inspection
Mode Always running as a service Ad-hoc investigation tool
Output Alerts (stdout, syslog, file, HTTP) Raw syscall events or filtered views
Rules YAML rule files with conditions Command-line filters
Use case “Alert me when X happens” “Show me what happened”

Use Falco for continuous monitoring and alerting.
Use sysdig for incident investigation after a Falco alert or to understand a container’s behavior.

Question 4

Write a sysdig command to watch all process executions inside a container named web-server.

Show Answer 
sudo sysdig "evt.type=execve and container.name=web-server"

For more detail including the full command line:

sudo sysdig -p "%evt.time %proc.name %proc.cmdline" \
  "evt.type=execve and container.name=web-server"

To also see the parent process:

sudo sysdig -p "%evt.time %proc.pname → %proc.name %proc.cmdline" \
  "evt.type=execve and container.name=web-server"

Question 5

Why does Falco need to run with elevated privileges (either as root or with the kernel module loaded)?

Show Answer

Falco intercepts kernel-level syscalls, which requires privileged access:

  1. Kernel module — loading a kernel module (insmod) requires root privileges. The module hooks into the syscall table.
  2. eBPF probe — loading BPF programs requires CAP_SYS_ADMIN or CAP_BPF capability. The probe attaches to kernel tracepoints.

Without these privileges, Falco cannot see any syscalls and is useless. When deployed as a DaemonSet, the Falco pods run with privileged: true and hostPID: true for this reason.

Next: Immutable Containers →

Updated:

You may also enjoy