Amazon EKS — DaemonSets and System Workloads

DaemonSets ensure a copy of a pod runs on every node (or a subset). They’re essential for node-level agents: log collectors, monitoring agents, security tools, and storage drivers.

---

DaemonSet Basics

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentbit
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: fluentbit
  template:
    metadata:
      labels:
        app: fluentbit
    spec:
      serviceAccountName: fluentbit
      containers:
        - name: fluentbit
          image: public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              memory: "256Mi"
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: containers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: containers
          hostPath:
            path: /var/lib/docker/containers
      tolerations:
        - operator: Exists     # Run on all nodes including tainted ones

---

Common DaemonSets on EKS

DaemonSet	Purpose	Installed By
aws-node (VPC CNI)	Pod networking	EKS add-on (pre-installed)
kube-proxy	Service networking	EKS add-on (pre-installed)
ebs-csi-node	EBS volume mounting	EBS CSI Driver add-on
efs-csi-node	EFS volume mounting	EFS CSI Driver add-on
fluent-bit	Log shipping to CloudWatch	AWS for Fluent Bit
cloudwatch-agent	Metrics collection	CloudWatch agent
node-exporter	Prometheus metrics	kube-prometheus-stack
falco	Runtime security	Falco

---

Targeting Specific Nodes

Node Selector

spec:
  template:
    spec:
      nodeSelector:
        node-type: gpu     # Only run on GPU nodes

Node Affinity

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: eks.amazonaws.com/nodegroup
                    operator: In
                    values:
                      - monitoring-nodes

---

Tolerations for Tainted Nodes

If nodes have taints (e.g., dedicated GPU nodes), DaemonSets need tolerations:

spec:
  template:
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
        - key: "dedicated"
          operator: "Equal"
          value: "monitoring"
          effect: "NoSchedule"

To run on all nodes including any tainted nodes:

tolerations:
  - operator: Exists

---

DaemonSets and Fargate

DaemonSets do NOT run on Fargate. There is no “node” to target — Fargate pods run on individual micro-VMs.

If a DaemonSet provides functionality you need on Fargate (e.g., log shipping), use a sidecar container instead:

# Sidecar pattern for Fargate pods
spec:
  containers:
    - name: app
      image: my-app:v1
    - name: log-shipper
      image: public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
      # Shares the app's log volume

Alternatively, Fargate has built-in log routing via its own Fluent Bit configuration. See Observability.

---

EKS System DaemonSets

EKS runs critical DaemonSets in kube-system:

aws-node (VPC CNI)

kubectl get daemonset aws-node -n kube-system
kubectl describe daemonset aws-node -n kube-system

Key environment variables:

Variable	Purpose
WARM_IP_TARGET	Number of warm IPs to maintain per node
MINIMUM_IP_TARGET	Minimum IPs to keep allocated
ENABLE_PREFIX_DELEGATION	Enable /28 prefix delegation
AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG	Enable custom networking

kube-proxy

kubectl get daemonset kube-proxy -n kube-system

Manages iptables/IPVS rules for Service routing on each node.

---

Updating DaemonSets

Rolling Update (Default)

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1     # Update one node at a time

OnDelete

Manual control — pods update only when you delete them:

spec:
  updateStrategy:
    type: OnDelete

---

Priority Classes for System Workloads

Ensure critical DaemonSets aren’t evicted:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: system-critical
value: 1000000
globalDefault: false
description: "For critical system DaemonSets"
---
spec:
  template:
    spec:
      priorityClassName: system-critical

EKS also provides built-in priority classes:

Priority Class	Value	Use For
system-node-critical	2000001000	Node-level agents (VPC CNI, kube-proxy)
system-cluster-critical	2000000000	Cluster-level components (CoreDNS)

---

Best Practices

1. Set resource requests and limits — DaemonSets compete with app pods for node resources
2. Use tolerations carefully — operator: Exists runs everywhere, including masters
3. Monitor DaemonSet pod counts — mismatches indicate scheduling problems
4. Use maxUnavailable: 1 for rolling updates — preserves node coverage
5. Remember Fargate limitations — use sidecars instead of DaemonSets
6. Assign priority classes to critical system DaemonSets

---