KCSA — Node & OS Hardening
Why Node Security Matters
Every Kubernetes workload runs on a node — a physical or virtual machine running a container runtime. If an attacker compromises a node, they gain access to:
- All pods scheduled on that node
- All Secrets mounted into those pods
- The kubelet and its credentials
- The container runtime (potentially allowing container escape)
- The host filesystem (including etcd data if it is a control plane node)
A single compromised node can be the starting point for lateral movement across the entire cluster. This is why node hardening is the foundation of platform security.
Minimal OS Distributions
Traditional Linux distributions (Ubuntu Server, CentOS) come with many packages, services, and tools pre-installed. Each additional package is a potential attack surface. Minimal OS distributions are purpose-built for running containers and nothing else.
| Distribution | Vendor | Key Features | Update Model |
|---|---|---|---|
| Bottlerocket | AWS | Immutable, API-driven config, no SSH by default | Atomic updates |
| Flatcar Container Linux | Kinvolk/Microsoft | Immutable, auto-updates, systemd-based | Rolling updates |
| Talos Linux | Sidero Labs | Immutable, no shell, API-only management | In-place upgrades |
| Ubuntu Minimal | Canonical | Reduced package set, familiar tooling | APT updates |
| Google Container-Optimized OS | Read-only rootfs, verified boot | Automatic |
Why Minimal OS Matters
- Smaller attack surface — fewer packages means fewer vulnerabilities
- Immutable infrastructure — OS cannot be modified at runtime, preventing persistence
- Automated updates — security patches are applied without manual intervention
- No unnecessary services — no SSH daemon, no package manager (in extreme cases like Talos)
For the KCSA exam, know that minimal OS distributions exist and why they are preferred for Kubernetes nodes. You will not be asked to install or configure one.
CIS Kubernetes Benchmark
The Center for Internet Security (CIS) publishes a comprehensive benchmark for Kubernetes. It is the industry-standard guide for hardening Kubernetes clusters and is heavily referenced in the KCSA exam.
The CIS Kubernetes Benchmark covers:
- Control Plane Components — API server, controller manager, scheduler, etcd
- Worker Nodes — kubelet, container runtime, kube-proxy
- Policies — RBAC, Pod Security, Network Policies, Secrets management
- Managed Services — Additional checks for EKS, AKS, GKE
CIS Benchmark Categories
| Category | What It Covers | Example Checks |
|---|---|---|
| Control Plane Configuration | API server flags, etcd encryption | --anonymous-auth=false, --encryption-provider-config set |
| etcd | Peer/client TLS, access control | Client cert auth enabled, peer TLS configured |
| Control Plane Node | File permissions, process arguments | Manifest file permissions 600, correct ownership |
| Worker Node | Kubelet configuration, file permissions | --anonymous-auth=false on kubelet, --read-only-port=0 |
| Policies | RBAC, PSA, NetworkPolicy | No wildcard RBAC rules, PSA labels on namespaces |
| Logging & Auditing | Audit policy configuration | Audit log path configured, audit policy covers key resources |
CIS Benchmark Levels
Each recommendation is classified into one of two levels:
- Level 1 — Basic security hygiene that can be applied broadly without significant operational impact
- Level 2 — Defense-in-depth recommendations that may have operational trade-offs
The KCSA exam may ask about the purpose of the CIS Benchmark and what it covers. You do not need to memorize individual control IDs.
Node Hardening Checklist
The following practices apply to all Kubernetes nodes — control plane and worker — regardless of the OS distribution.
1. Disable Swap
Kubernetes requires swap to be disabled on nodes. The kubelet will refuse to start if swap is detected (unless explicitly allowed with --fail-swap-on=false).
# Disable swap immediately
sudo swapoff -a
# Remove swap entries from /etc/fstab to persist across reboots
sudo sed -i '/swap/d' /etc/fstab
Swap interferes with Kubernetes memory management and can cause unpredictable pod behavior.
2. Remove Unnecessary Packages
Strip down the node to only what is needed to run Kubernetes and the container runtime.
# Example: Remove unnecessary packages on Ubuntu
sudo apt purge -y telnet ftp rsh-client
sudo apt autoremove -y
3. Configure Firewall Rules
Only allow traffic that Kubernetes components need. Block everything else.
# Allow kubelet API (worker nodes)
sudo iptables -A INPUT -p tcp --dport 10250 -j ACCEPT
# Allow kube-proxy
sudo iptables -A INPUT -p tcp --dport 10256 -j ACCEPT
# Allow NodePort range
sudo iptables -A INPUT -p tcp --dport 30000:32767 -j ACCEPT
# Drop everything else
sudo iptables -A INPUT -j DROP
4. SSH Hardening
If SSH is required on nodes, harden it significantly:
# /etc/ssh/sshd_config
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2
AllowUsers k8s-admin
Ideally, disable SSH entirely and use API-driven node management (as with Bottlerocket or Talos).
5. Automatic Security Updates
Configure unattended upgrades for security patches:
# Ubuntu — install and enable unattended-upgrades
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
6. Filesystem Permissions
Secure critical Kubernetes files on the node:
| File / Directory | Recommended Permission | Owner |
|---|---|---|
/etc/kubernetes/manifests/ |
600 |
root:root |
/etc/kubernetes/pki/ |
600 |
root:root |
/var/lib/kubelet/config.yaml |
644 |
root:root |
/etc/cni/net.d/ |
644 |
root:root |
/var/lib/etcd/ |
700 |
etcd:etcd |
Kernel Security Features
Linux provides several kernel-level security mechanisms that Kubernetes leverages:
Linux Security Modules (LSMs)
| Module | Description | Usage in Kubernetes |
|---|---|---|
| AppArmor | Restricts program capabilities per profile | Container-level profiles via annotations |
| SELinux | Mandatory Access Control using labels and policies | Pod securityContext.seLinuxOptions |
| Seccomp | Filters system calls available to a process | Pod securityContext.seccompProfile |
Namespaces and cgroups
These are not the same as Kubernetes Namespaces — they are Linux kernel features:
- Linux namespaces — Isolate process views (PID, network, mount, UTS, IPC, user)
- cgroups — Limit and account for resource usage (CPU, memory, I/O)
Together, namespaces and cgroups form the foundation of container isolation. However, containers still share the host kernel — which is why sandboxed runtimes (covered in the next topic) exist.
kube-bench — Automated CIS Benchmark Checking
kube-bench is an open-source tool by Aqua Security that checks whether your Kubernetes cluster is deployed according to CIS Benchmark recommendations.
What kube-bench Does
- Runs checks against control plane nodes and worker nodes
- Reports results as PASS, FAIL, WARN, or INFO
- Provides remediation steps for each failed check
- Supports CIS Benchmarks for vanilla Kubernetes, EKS, AKS, and GKE
Running kube-bench
# Run on a control plane node
kube-bench run --targets master
# Run on a worker node
kube-bench run --targets node
# Run all checks
kube-bench run
Example Output
[INFO] 1 Control Plane Security Configuration
[INFO] 1.1 Control Plane Node Configuration Files
[PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive
[FAIL] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root
[WARN] 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive
== Remediations master ==
1.1.2 Run: chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml
== Summary master ==
15 checks PASS
2 checks FAIL
3 checks WARN
0 checks INFO
kube-bench is a detection tool, not a remediation tool. It tells you what is wrong but does not fix it automatically.
Node Auto-Upgrade Strategies
Keeping nodes up to date is critical for security but must be done without disrupting workloads.
| Strategy | How It Works | Disruption | Use Case |
|---|---|---|---|
| Rolling upgrade | Cordon, drain, upgrade, uncordon one node at a time | Minimal | Production clusters |
| Blue-green nodes | Create new node group, migrate workloads, delete old nodes | None | Zero-downtime requirement |
| In-place upgrade | Upgrade node OS/kubelet without draining | Possible | Non-critical environments |
| Managed auto-upgrade | Cloud provider handles upgrades (EKS, AKS, GKE managed node groups) | Managed | Cloud-managed clusters |
The Drain and Cordon Process
# Mark node as unschedulable (no new pods)
kubectl cordon node-1
# Evict all pods from the node
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
# Perform upgrade (OS patches, kubelet upgrade, etc.)
# ...
# Mark node as schedulable again
kubectl uncordon node-1
Always ensure PodDisruptionBudgets (PDBs) are configured so that draining a node does not take your application below the required number of replicas.
Key Takeaways for the Exam
| Concept | What to Remember |
|---|---|
| Node compromise | Compromising a node exposes all pods, Secrets, and kubelet credentials on it |
| Minimal OS | Bottlerocket, Flatcar, Talos — reduced attack surface, immutable, auto-updating |
| CIS Benchmark | Industry-standard hardening guide covering control plane, workers, and policies |
| kube-bench | Open-source tool that automates CIS Benchmark checking — reports PASS/FAIL/WARN |
| Node hardening | Disable swap, remove packages, harden SSH, configure firewall, secure file permissions |
| Kernel security | AppArmor, SELinux, Seccomp — restrict what processes can do at the kernel level |
| Node upgrades | Rolling upgrades with cordon/drain/uncordon — PDBs prevent disruption |
- Node security is foundational — everything else depends on it
- The KCSA exam tests whether you know what to harden and why, not how to write configuration files
- kube-bench + CIS Benchmarks is the most commonly tested combination in this area
