ELK Stack — Filebeat

6 minute read

What Is Filebeat?

Filebeat is a lightweight log shipper that monitors log files or locations you specify, collects log events, and forwards them to Elasticsearch or Logstash for indexing. It is the most widely deployed Beat and serves as the primary tool for log collection in the Elastic Stack.

Filebeat is designed to be reliable and low-latency. It uses a registry to track the read position of each file, ensuring that no log line is lost — even if the output (Elasticsearch or Logstash) is temporarily unavailable.

Filebeat Architecture

┌──────────────────────────────────────────────────────────┐
│                        Filebeat                           │
│                                                           │
│  ┌─────────────────────────────────────────────────────┐ │
│  │                    Inputs                            │ │
│  │                                                      │ │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │ │
│  │  │  /var/log/   │  │ /app/logs/  │  │  container   │ │ │
│  │  │  syslog      │  │  app.log    │  │  stdout      │ │ │
│  │  └──────┬───────┘  └──────┬──────┘  └──────┬──────┘ │ │
│  │         │                 │                 │        │ │
│  └─────────┼─────────────────┼─────────────────┼────────┘ │
│            │                 │                 │          │
│            ▼                 ▼                 ▼          │
│  ┌─────────────────────────────────────────────────────┐ │
│  │               Harvesters (one per file)              │ │
│  │     Read each file line by line, track offset        │ │
│  └──────────────────────┬──────────────────────────────┘ │
│                         │                                 │
│                         ▼                                 │
│  ┌─────────────────────────────────────────────────────┐ │
│  │                     Spooler                          │ │
│  │         Aggregates events into batches               │ │
│  └──────────────────────┬──────────────────────────────┘ │
│                         │                                 │
│                         ▼                                 │
│  ┌─────────────────────────────────────────────────────┐ │
│  │           Processors (optional)                      │ │
│  │    Enrich, filter, transform before shipping         │ │
│  └──────────────────────┬──────────────────────────────┘ │
│                         │                                 │
│                         ▼                                 │
│  ┌─────────────────────────────────────────────────────┐ │
│  │                    Output                            │ │
│  │     Elasticsearch │ Logstash │ Kafka │ File          │ │
│  └─────────────────────────────────────────────────────┘ │
│                                                           │
│  ┌─────────────────────────────────────────────────────┐ │
│  │  Registry  (data/registry/filebeat/log.json)        │ │
│  │  Tracks file offsets for at-least-once delivery      │ │
│  └─────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘

Key Components

Component	Description
Input	Defines the source of log data (file paths, container logs, stdin)
Harvester	One harvester per file — reads content line by line and sends to the spooler
Spooler	Aggregates events and batches them before sending to the output
Registry	Persists the last read offset for each file — survives restarts
Output	Destination for collected events (Elasticsearch, Logstash, Kafka, etc.)

Installation

# Debian/Ubuntu
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.17.0-amd64.deb
sudo dpkg -i filebeat-8.17.0-amd64.deb

# RHEL/CentOS
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.17.0-x86_64.rpm
sudo rpm -vi filebeat-8.17.0-x86_64.rpm

# macOS (Homebrew)
brew install elastic/tap/filebeat-full

# Docker
docker pull docker.elastic.co/beats/filebeat:8.17.0

Complete filebeat.yml Configuration

Below is a comprehensive filebeat.yml that covers the most common settings:

# =========================== Filebeat Inputs ============================
filebeat.inputs:

  # --- Log input: collect application logs ---
  - type: log
    id: app-logs
    enabled: true
    paths:
      - /var/log/myapp/*.log
      - /opt/app/logs/application.log
    # Exclude lines that match a pattern
    exclude_lines: ['^DEBUG']
    # Include only lines that match a pattern
    include_lines: ['^ERR', '^WARN', '^INFO']
    # Exclude files matching a pattern
    exclude_files: ['\.gz$']
    # Add custom fields to every event from this input
    fields:
      app: myapp
      environment: production
    fields_under_root: true
    # Tags for filtering in Logstash/ES
    tags: ["app-logs", "production"]

    # --- Multiline settings ---
    multiline.type: pattern
    multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    multiline.negate: true
    multiline.match: after
    multiline.max_lines: 50
    multiline.timeout: 5s

  # --- Container input: collect Docker/K8s container logs ---
  - type: container
    id: docker-logs
    enabled: true
    paths:
      - /var/lib/docker/containers/*/*.log
    # Parse container log format (docker-json, cri)
    stream: all          # stdout, stderr, or all
    format: docker
    fields:
      source: docker
    fields_under_root: true

  # --- Stdin input (useful for testing/piping) ---
  - type: stdin
    id: stdin-input
    enabled: false

# =========================== Filebeat Modules ============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

# ============================== Processors ===============================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - drop_event:
      when:
        regexp:
          message: "^\\s*$"

# ========================= Elasticsearch Output ==========================
output.elasticsearch:
  hosts: ["https://es-node-1:9200", "https://es-node-2:9200"]
  protocol: "https"
  username: "filebeat_writer"
  password: "changeme"
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
    certificate: "/etc/filebeat/certs/filebeat.crt"
    key: "/etc/filebeat/certs/filebeat.key"
  # Bulk indexing settings
  bulk_max_size: 2048
  compression_level: 3

# ========================== Logstash Output ==============================
# output.logstash:
#   hosts: ["logstash-host:5044"]
#   ssl:
#     certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
#     certificate: "/etc/filebeat/certs/filebeat.crt"
#     key: "/etc/filebeat/certs/filebeat.key"
#   loadbalance: true
#   bulk_max_size: 2048

# ============================ Kafka Output ===============================
# output.kafka:
#   hosts: ["kafka-1:9092", "kafka-2:9092", "kafka-3:9092"]
#   topic: "filebeat-logs"
#   partition.round_robin:
#     reachable_only: true
#   required_acks: 1
#   compression: gzip
#   max_message_bytes: 1000000

# ============================== Kibana ===================================
setup.kibana:
  host: "https://kibana-host:5601"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/filebeat/certs/ca.crt"]

# ============================= Dashboards ================================
setup.dashboards.enabled: true

# ========================== ILM (Index Lifecycle) ========================
setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "filebeat"

# ============================== Logging ==================================
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
  rotateeverybytes: 10485760    # 10 MB
logging.metrics.enabled: true
logging.metrics.period: 30s

# ============================== Keystore =================================
# Store sensitive values (passwords, API keys) in the Filebeat keystore
# instead of plaintext in filebeat.yml:
#
#   filebeat keystore create
#   filebeat keystore add ES_PASSWORD
#   filebeat keystore add KAFKA_PASSWORD
#
# Reference in config:
#   password: "${ES_PASSWORD}"

Input Types

Filebeat supports multiple input types for different data sources:

Input Type	Use Case
`log`	Tail regular log files on disk
`container`	Collect Docker or Kubernetes container logs
`stdin`	Read from standard input (testing/piping)
`tcp`	Listen for log events over TCP
`udp`	Listen for log events over UDP
`syslog`	Receive syslog messages (RFC 3164 / RFC 5424)
`journald`	Collect systemd journal entries
`aws-s3`	Read logs from AWS S3 buckets

Multiline Settings

Many applications produce log entries that span multiple lines (Java stack traces, JSON payloads). Use multiline settings to merge them into a single event.

# Example: Java stack trace — lines not starting with a timestamp
# are appended to the previous line
filebeat.inputs:
  - type: log
    paths:
      - /var/log/java-app/*.log
    multiline.type: pattern
    multiline.pattern: '^\d{4}-\d{2}-\d{2}[T ]'
    multiline.negate: true
    multiline.match: after
    multiline.max_lines: 100
    multiline.timeout: 10s

Pattern logic:

negate: true + match: after → Lines that do not match the pattern are appended to the previous line
negate: false + match: before → Lines that do match are prepended to the next line

Harvester and Registry

The harvester is the core reading mechanism. One harvester is started per file. It opens the file, reads it line by line, and sends content to the output.

The registry file (data/registry/filebeat/log.json) stores the state of every file Filebeat is tracking:

[
  {
    "source": "/var/log/myapp/app.log",
    "offset": 28456,
    "timestamp": "2026-04-01T14:44:00.000Z",
    "ttl": -1,
    "type": "log",
    "FileStateOS": {
      "inode": 1234567,
      "device": 64768
    }
  }
]

If Filebeat restarts, it reads the registry and resumes from the last known offset — ensuring no log lines are duplicated or lost.

Filebeat Keystore

Store sensitive values securely instead of putting them in plaintext config:

# Create the keystore
filebeat keystore create

# Add a secret
filebeat keystore add ES_PASSWORD
# (You will be prompted to enter the value)

# List stored keys
filebeat keystore list

# Remove a key
filebeat keystore remove ES_PASSWORD

Reference keystore values in filebeat.yml:

output.elasticsearch:
  hosts: ["https://es-node-1:9200"]
  username: "filebeat_writer"
  password: "${ES_PASSWORD}"

Starting and Managing Filebeat

# Test configuration
filebeat test config -e

# Test output connectivity
filebeat test output

# Start Filebeat (systemd)
sudo systemctl start filebeat
sudo systemctl enable filebeat

# Run in foreground (debug)
filebeat -e -d "*"

# Run with a specific config
filebeat -e -c /etc/filebeat/custom.yml

Quick Reference

Task	Command / Config
Install (Debian)	`sudo dpkg -i filebeat-8.17.0-amd64.deb`
Config file	`/etc/filebeat/filebeat.yml`
Data/registry	`/var/lib/filebeat/registry/`
Test config	`filebeat test config -e`
Test output	`filebeat test output`
Start service	`sudo systemctl start filebeat`
Enable on boot	`sudo systemctl enable filebeat`
View logs	`journalctl -u filebeat -f`
Keystore create	`filebeat keystore create`
Add secret	`filebeat keystore add KEY_NAME`

Next: Filebeat Modules — pre-built configurations for common log formats.

Share on

Twitter Facebook LinkedIn