ELK Stack — Fluentd and Fluent Bit

Overview

While Logstash and Filebeat are the native Elastic stack shippers, two CNCF projects — Fluentd and Fluent Bit — are widely used as alternatives, especially in Kubernetes environments. Together with Elasticsearch and Kibana, they form the EFK stack (Elasticsearch + Fluentd/Fluent Bit + Kibana).

---

Fluentd

Fluentd is an open-source data collector originally created by Treasure Data. It is a CNCF Graduated project — the same maturity level as Kubernetes itself.

Key characteristics:

• Written in Ruby with performance-critical parts in C
• Plugin ecosystem — 1,000+ community plugins for inputs, filters, and outputs
• Acts as a log aggregator (similar role to Logstash)
• Handles routing, buffering, and retries
• Higher memory footprint (~40-100 MB)

┌────────────┐     ┌────────────┐     ┌───────────────┐
│ Log Sources│────▶│  Fluentd   │────▶│ Elasticsearch │
│ (files,    │     │ (aggregate,│     │               │
│  syslog,   │     │  filter,   │     └───────────────┘
│  HTTP)     │     │  route)    │
└────────────┘     └────────────┘

---

Fluent Bit

Fluent Bit is the lightweight sibling of Fluentd. It is also a CNCF Graduated project and is purpose-built for high-throughput, low-resource environments like Kubernetes nodes and edge devices.

Key characteristics:

• Written in C — extremely fast and memory efficient
• Minimal footprint (~1-5 MB memory)
• Built-in Kubernetes filter for metadata enrichment
• Ideal as a node-level log shipper (similar role to Filebeat)
• Can forward to Fluentd for aggregation, or directly to Elasticsearch

---

The EFK Stack

The EFK stack replaces Logstash/Filebeat with Fluentd/Fluent Bit:

                     ELK Stack              EFK Stack
                   ┌───────────┐          ┌────────────┐
   Ship logs       │ Filebeat  │          │ Fluent Bit │
                   └─────┬─────┘          └──────┬─────┘
                         │                       │
   Process/Route   ┌─────▼─────┐          ┌──────▼─────┐
                   │ Logstash  │          │  Fluentd   │
                   └─────┬─────┘          └──────┬─────┘
                         │                       │
   Store           ┌─────▼──────────────────▶────▼─────┐
                   │         Elasticsearch              │
                   └─────┬──────────────────────┬──────┘
                         │                      │
   Visualize       ┌─────▼─────┐          ┌─────▼─────┐
                   │  Kibana   │          │  Kibana   │
                   └───────────┘          └───────────┘

Note: In many Kubernetes deployments, Fluent Bit ships logs directly to Elasticsearch — skipping the aggregation layer entirely.

---

Fluent Bit DaemonSet Deployment

ConfigMap — Fluent Bit Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Daemon        Off
        Log_Level     info
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

    [OUTPUT]
        Name            es
        Match           *
        Host            elasticsearch
        Port            9200
        Logstash_Format On
        Logstash_Prefix k8s-logs
        Retry_Limit     False
        Replace_Dots    On
        tls             Off
        Suppress_Type_Name On

  parsers.conf: |
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        json
        Format      json
        Time_Key    time
        Time_Format %d/%b/%Y:%H:%M:%S %z

DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      terminationGracePeriodSeconds: 10
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:3.0
          ports:
            - containerPort: 2020
              name: http-metrics
          volumeMounts:
            - name: config
              mountPath: /fluent-bit/etc/
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: varlogcontainers
              mountPath: /var/log/containers
              readOnly: true
            - name: varlogpods
              mountPath: /var/log/pods
              readOnly: true
          resources:
            limits:
              memory: 100Mi
              cpu: 200m
            requests:
              memory: 50Mi
              cpu: 50m
      volumes:
        - name: config
          configMap:
            name: fluent-bit-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  - apiGroups: [""]
    resources: ["namespaces", "pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logging

# Deploy
kubectl create namespace logging
kubectl apply -f fluent-bit.yaml

# Verify
kubectl get daemonset fluent-bit -n logging

# Check logs
kubectl logs -l app=fluent-bit -n logging --tail=20

---

Fluent Bit Configuration Explained

INPUT Section

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            cri
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

Parameter	Description
Name tail	Reads files like tail -f
Tag kube.*	Tags events for routing
Path	Container log files
Parser cri	Parses CRI log format
DB	Tracks file read position (like Filebeat registry)
Mem_Buf_Limit	Maximum memory per input
Skip_Long_Lines	Skip lines > buffer size

FILTER Section

[FILTER]
    Name                kubernetes
    Match               kube.*
    Merge_Log           On
    K8S-Logging.Parser  On

The kubernetes filter enriches each log event with pod metadata — identical in purpose to Filebeat’s add_kubernetes_metadata processor.

OUTPUT Section

[OUTPUT]
    Name            es
    Match           *
    Host            elasticsearch
    Port            9200
    Logstash_Format On
    Logstash_Prefix k8s-logs

Sends all matched events to Elasticsearch using the es output plugin. Logstash_Format On creates daily indices like k8s-logs-2026.04.01.

---

Comparison — Fluentd vs Fluent Bit vs Logstash vs Filebeat

Feature	Filebeat	Fluent Bit	Logstash	Fluentd
Role	Shipper	Shipper	Processor	Processor / Aggregator
Language	Go	C	Java (JRuby)	Ruby + C
Memory	~30-50 MB	~1-5 MB	~200-500 MB	~40-100 MB
CPU usage	Low	Very low	High	Medium
Plugin ecosystem	Modules + inputs	~100 plugins	~200 plugins	~1,000 plugins
K8s metadata	✅ (processor)	✅ (built-in filter)	✅ (filter plugin)	✅ (plugin)
CNCF project	❌ (Elastic)	✅ Graduated	❌ (Elastic)	✅ Graduated
Best for	Elastic-native shipping	Lightweight K8s shipping	Complex pipelines	Log aggregation
K8s deployment	DaemonSet	DaemonSet	Deployment	Deployment/DaemonSet
Config format	YAML	INI-like (key-value)	Custom DSL	Ruby-like DSL
Output to ES	Native	Plugin	Native	Plugin

---

When to Use Each

Use Filebeat when:

• You are fully invested in the Elastic ecosystem
• You want autodiscover with pod annotations
• You need Elastic modules (pre-built parsers for nginx, MySQL, etc.)

Use Fluent Bit when:

• You need the lowest possible resource footprint
• You prefer CNCF-standard tooling
• Your cluster has tight resource budgets (edge, IoT, small clusters)

Use Logstash when:

• You need complex transformations (grok, enrichment, conditional routing)
• You have a centralized processing pipeline

Use Fluentd when:

• You need to fan-out to multiple destinations (S3, Kafka, ES, etc.)
• You need a specific Fluentd community plugin
• You want a CNCF-standard aggregation layer

Common Combinations

# Combination 1: Pure Elastic
Filebeat (DaemonSet) → Logstash (Deployment) → Elasticsearch

# Combination 2: Pure CNCF
Fluent Bit (DaemonSet) → Fluentd (Deployment) → Elasticsearch

# Combination 3: Lightweight
Fluent Bit (DaemonSet) → Elasticsearch (direct)

# Combination 4: Hybrid
Fluent Bit (DaemonSet) → Logstash (Deployment) → Elasticsearch

---

Quick Reference

Concept	Details
Fluent Bit role	Lightweight log shipper (like Filebeat)
Fluentd role	Log aggregator/router (like Logstash)
EFK stack	Elasticsearch + Fluentd/Fluent Bit + Kibana
Fluent Bit image	fluent/fluent-bit:3.0
Config format	INI-like key-value sections
K8s filter	[FILTER] Name kubernetes
ES output	[OUTPUT] Name es
Memory usage	Fluent Bit ~1-5 MB, Fluentd ~40-100 MB
CNCF status	Both Graduated projects

---

Next: Log Retention in Kubernetes — design ILM policies and storage strategies for Kubernetes logs.