ELK Stack — Filebeat DaemonSet

Deploying Filebeat as a DaemonSet

A DaemonSet ensures exactly one Filebeat pod runs on every node in the cluster. Each Filebeat pod mounts the host’s log directory, reads container log files, enriches them with Kubernetes metadata, and ships them to Elasticsearch.

This is the most common — and simplest — approach for centralized Kubernetes logging.

---

Architecture Overview

   Node 1                 Node 2                 Node 3
┌────────────┐        ┌────────────┐        ┌────────────┐
│ Pod A      │        │ Pod C      │        │ Pod E      │
│ Pod B      │        │ Pod D      │        │ Pod F      │
│            │        │            │        │            │
│ Filebeat   │        │ Filebeat   │        │ Filebeat   │
│ (DaemonSet)│        │ (DaemonSet)│        │ (DaemonSet)│
└─────┬──────┘        └─────┬──────┘        └─────┬──────┘
      │                     │                     │
      └──────────┬──────────┴──────────┬──────────┘
                 ▼                     ▼
          ┌─────────────┐    ┌──────────────┐
          │Elasticsearch│    │   Kibana     │
          └─────────────┘    └──────────────┘

---

Complete DaemonSet Manifest

The full deployment requires five Kubernetes resources:

1. Namespace (optional)
2. ConfigMap — Filebeat configuration
3. ServiceAccount — identity for the Filebeat pods
4. ClusterRole + ClusterRoleBinding — RBAC permissions
5. DaemonSet — the Filebeat pods

1. ConfigMap — Filebeat Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: elastic-system
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
                - logs_path:
                    logs_path: "/var/log/containers/"

    output.elasticsearch:
      hosts: ["http://elasticsearch:9200"]
      index: "k8s-logs-%{+yyyy.MM.dd}"

    setup.template.name: "k8s-logs"
    setup.template.pattern: "k8s-logs-*"
    setup.ilm.enabled: false

    logging.level: info
    logging.to_stderr: true

2. ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: elastic-system
  labels:
    app: filebeat

3. ClusterRole — RBAC Permissions

Filebeat needs cluster-wide read access to pods, namespaces, and nodes to enrich logs with Kubernetes metadata.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - nodes
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - replicasets
    verbs: ["get", "list", "watch"]

4. ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
  labels:
    app: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: elastic-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

5. DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: elastic-system
  labels:
    app: filebeat
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:8.13.0
          args: ["-c", "/etc/filebeat/filebeat.yml", "-e"]
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: ELASTICSEARCH_HOST
              value: "elasticsearch"
            - name: ELASTICSEARCH_PORT
              value: "9200"
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
              cpu: 200m
            requests:
              memory: 100Mi
              cpu: 100m
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat
              readOnly: true
            - name: varlogcontainers
              mountPath: /var/log/containers
              readOnly: true
            - name: varlogpods
              mountPath: /var/log/pods
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
      volumes:
        - name: config
          configMap:
            name: filebeat-config
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate

---

Deploying the Manifests

# Create namespace
kubectl create namespace elastic-system

# Apply all manifests (if saved in one file)
kubectl apply -f filebeat-k8s.yaml

# Or apply individually
kubectl apply -f filebeat-configmap.yaml
kubectl apply -f filebeat-serviceaccount.yaml
kubectl apply -f filebeat-clusterrole.yaml
kubectl apply -f filebeat-clusterrolebinding.yaml
kubectl apply -f filebeat-daemonset.yaml

# Verify DaemonSet is running on all nodes
kubectl get daemonset filebeat -n elastic-system

Expected output:

NAME       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
filebeat   3         3         3       3             3           <none>          45s

---

Filebeat Container Input

The container input type is purpose-built for Kubernetes. It understands the CRI log format and automatically handles multi-line detection.

filebeat.inputs:
  - type: container
    paths:
      - /var/log/containers/*.log
    # Exclude logs from kube-system namespace
    exclude_files:
      - '.*kube-system.*'
    # Parse JSON log content
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: message

---

Autodiscover with Kubernetes Provider

Autodiscover dynamically configures Filebeat inputs based on pod labels and annotations. This lets you apply different parsing rules to different applications without changing the Filebeat ConfigMap.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*-${data.container.id}.log
      templates:
        # Custom config for nginx pods
        - condition:
            contains:
              kubernetes.labels.app: "nginx"
          config:
            - type: container
              paths:
                - /var/log/containers/*-${data.container.id}.log
              processors:
                - dissect:
                    tokenizer: '%{client_ip} - - [%{timestamp}] "%{method} %{path} %{protocol}" %{status} %{bytes}'
                    field: "message"
                    target_prefix: "nginx"
        # Custom config for Java apps (multi-line stack traces)
        - condition:
            contains:
              kubernetes.labels.app: "java-api"
          config:
            - type: container
              paths:
                - /var/log/containers/*-${data.container.id}.log
              multiline.type: pattern
              multiline.pattern: '^\s+(at|\.{3}|Caused by)'
              multiline.negate: false
              multiline.match: after

Pod Annotations for Autodiscover Hints

With hints.enabled: true, Filebeat reads configuration from pod annotations:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    co.elastic.logs/enabled: "true"
    co.elastic.logs/multiline.pattern: '^\['
    co.elastic.logs/multiline.negate: "true"
    co.elastic.logs/multiline.match: "after"
    co.elastic.logs/processors.add_fields.target: "project"
    co.elastic.logs/processors.add_fields.fields.name: "checkout-service"
spec:
  containers:
    - name: app
      image: my-app:latest

---

Kubernetes Metadata Enrichment

The add_kubernetes_metadata processor is the key to making Kubernetes logs useful. It adds fields like pod name, namespace, labels, and node name to every log event.

processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
        - logs_path:
            logs_path: "/var/log/containers/"

This adds the following fields to each log event:

{
  "kubernetes": {
    "pod": {
      "name": "frontend-abc12",
      "uid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
    },
    "namespace": "production",
    "node": {
      "name": "worker-node-1"
    },
    "container": {
      "name": "nginx"
    },
    "labels": {
      "app": "frontend",
      "version": "v2.1"
    },
    "replicaset": {
      "name": "frontend-abc12"
    },
    "deployment": {
      "name": "frontend"
    }
  }
}

Now you can filter logs in Kibana by namespace, pod, deployment, or any Kubernetes label.

---

Volume Mounts Explained

Filebeat needs access to three paths on the host:

Volume Mount	Host Path	Purpose
/var/log/containers	/var/log/containers	Symlinks to container log files
/var/log/pods	/var/log/pods	Actual container log files
/usr/share/filebeat/data	/var/lib/filebeat-data	Filebeat registry (tracks read position)

The data volume is critical — it stores the registry file that tracks which log lines have already been shipped. Without it, Filebeat would re-ship all logs every time the pod restarts.

---

Filebeat Helm Chart

For production deployments, the official Helm chart simplifies installation and upgrades.

# Add the Elastic Helm repository
helm repo add elastic https://helm.elastic.co
helm repo update

# Install Filebeat
helm install filebeat elastic/filebeat \
  --namespace elastic-system \
  --create-namespace \
  --set daemonset.enabled=true \
  --set daemonset.filebeatConfig."filebeat\.yml"="$(cat filebeat-values.yml)"

# Check status
helm status filebeat -n elastic-system

# Upgrade with new config
helm upgrade filebeat elastic/filebeat \
  --namespace elastic-system \
  -f filebeat-values.yaml

---

Troubleshooting

# Check Filebeat pod logs
kubectl logs -l app=filebeat -n elastic-system --tail=50

# Check if Filebeat is connecting to Elasticsearch
kubectl logs -l app=filebeat -n elastic-system | grep -i "connection\|error\|elasticsearch"

# Exec into a Filebeat pod to inspect
kubectl exec -it $(kubectl get pods -l app=filebeat -n elastic-system -o jsonpath='{.items[0].metadata.name}') \
  -n elastic-system -- bash

# Inside the pod — check registry
cat /usr/share/filebeat/data/registry/filebeat/log.json | head -20

# Verify volume mounts
ls -la /var/log/containers/

---

Quick Reference

Concept	Details
Deployment type	DaemonSet (one pod per node)
Input type	container
Log path	/var/log/containers/*.log
Metadata processor	add_kubernetes_metadata
Dynamic config	Autodiscover with kubernetes provider
Pod-level config	Annotations (co.elastic.logs/*)
RBAC required	ClusterRole with get/list/watch on pods, namespaces, nodes
Data persistence	hostPath volume for registry
Helm chart	elastic/filebeat

---

Next: ECK — Elastic Cloud on Kubernetes — run Elasticsearch and Kibana natively inside your cluster.