ELK Stack — Filebeat Modules

4 minute read

What Are Filebeat Modules?

Modules are pre-packaged configurations that simplify collecting, parsing, and visualizing logs from common sources. Each module bundles:

  • Input configuration — where to find the log files
  • Ingest pipeline — Elasticsearch ingest node pipeline to parse and enrich log lines
  • Kibana dashboards — pre-built visualizations for quick analysis

Instead of writing custom Logstash filters or Filebeat processors, modules let you get up and running in minutes.

Available Modules

Filebeat ships with 50+ modules. Some of the most commonly used:

Module Collects logs from
system Syslog and auth logs (/var/log/syslog, /var/log/auth.log)
nginx Nginx access and error logs
apache Apache HTTP server access and error logs
elasticsearch Elasticsearch cluster logs
mysql MySQL slow query and error logs
postgresql PostgreSQL logs
redis Redis server logs
kafka Kafka broker logs
docker Docker container logs via json-file driver
auditd Linux audit daemon logs

Listing, Enabling, and Disabling Modules

# List all available modules and their status
filebeat modules list

# Enable a module
filebeat modules enable system nginx mysql

# Disable a module
filebeat modules disable apache

# Verify enabled modules
ls /etc/filebeat/modules.d/*.yml | grep -v disabled

When you enable a module, Filebeat renames the config file from <module>.yml.disabled to <module>.yml inside the modules.d/ directory.

The modules.d Directory

/etc/filebeat/modules.d/
├── apache.yml.disabled
├── auditd.yml.disabled
├── docker.yml.disabled
├── elasticsearch.yml.disabled
├── kafka.yml.disabled
├── mysql.yml
├── nginx.yml
├── postgresql.yml.disabled
├── redis.yml.disabled
└── system.yml

Files ending in .yml are enabled. Files ending in .yml.disabled are disabled.

System Module

The system module collects syslog and auth logs from a Linux system.

# /etc/filebeat/modules.d/system.yml
- module: system
  syslog:
    enabled: true
    # Default paths: /var/log/syslog (Debian) or /var/log/messages (RHEL)
    # Override if your logs are in a custom location:
    # var.paths: ["/var/log/syslog"]

  auth:
    enabled: true
    # Default paths: /var/log/auth.log (Debian) or /var/log/secure (RHEL)
    # var.paths: ["/var/log/auth.log"]

What it parses: timestamp, hostname, process name, PID, message, SSH login attempts, sudo commands.

Nginx Module

# /etc/filebeat/modules.d/nginx.yml
- module: nginx
  access:
    enabled: true
    # Default: /var/log/nginx/access.log*
    var.paths: ["/var/log/nginx/access.log*"]

  error:
    enabled: true
    # Default: /var/log/nginx/error.log*
    var.paths: ["/var/log/nginx/error.log*"]

  ingress_controller:
    enabled: false

Parsed fields: client IP, HTTP method, URI, status code, response size, user agent, referrer, upstream response time.

Apache Module

# /etc/filebeat/modules.d/apache.yml
- module: apache
  access:
    enabled: true
    var.paths: ["/var/log/apache2/access.log*"]

  error:
    enabled: true
    var.paths: ["/var/log/apache2/error.log*"]

Parsed fields: client IP, identity, user, timestamp, method, URL, HTTP version, response code, bytes sent.

Elasticsearch Module

# /etc/filebeat/modules.d/elasticsearch.yml
- module: elasticsearch
  server:
    enabled: true
    var.paths: ["/var/log/elasticsearch/*.log"]

  slowlog:
    enabled: true
    var.paths: ["/var/log/elasticsearch/*_index_search_slowlog.log"]

  deprecation:
    enabled: true
    var.paths: ["/var/log/elasticsearch/*_deprecation.log"]

  gc:
    enabled: true
    var.paths: ["/var/log/elasticsearch/gc.log.[0-9]*"]

  audit:
    enabled: false

MySQL Module

# /etc/filebeat/modules.d/mysql.yml
- module: mysql
  error:
    enabled: true
    var.paths: ["/var/log/mysql/error.log*"]

  slowlog:
    enabled: true
    var.paths: ["/var/log/mysql/mysql-slow.log*"]
    # Set custom slow query threshold (seconds)
    # var.slow_query_log_type: "traditional"

Parsed fields: timestamp, thread ID, user, host, query time, lock time, rows sent, rows examined, SQL query.

Custom Module Configuration with Variables

Modules support variables (var.*) that let you customize behavior without editing the module code:

- module: nginx
  access:
    enabled: true
    # Override log file paths
    var.paths: ["/opt/nginx/logs/access.log"]
    # Override input type (default: log)
    # var.input: "docker"
    # Custom tags
    var.tags: ["nginx", "webserver", "production"]

Common module variables:

Variable Description
var.paths Override default log file paths
var.tags Add custom tags to events
var.input Change input type (log, docker, tcp, etc.)
var.convert_timezone Convert timestamps to UTC

Setting Up Module Dashboards

Modules come with pre-built Kibana dashboards. Load them with:

# Load dashboards for all enabled modules
filebeat setup --dashboards

# Load dashboards and connect to a specific Kibana
filebeat setup --dashboards \
  -E setup.kibana.host="https://kibana:5601" \
  -E setup.kibana.ssl.enabled=true

# Load only index templates and ILM policies (no dashboards)
filebeat setup --index-management

# Load only ingest pipelines
filebeat setup --pipelines

After loading, open Kibana and navigate to Dashboard → search for [Filebeat Nginx] or [Filebeat System].

Complete Workflow Example

Set up Filebeat with system and Nginx modules from scratch:

# 1. Install Filebeat
sudo apt-get install filebeat

# 2. Enable desired modules
sudo filebeat modules enable system nginx

# 3. Configure modules (edit paths if needed)
sudo vi /etc/filebeat/modules.d/system.yml
sudo vi /etc/filebeat/modules.d/nginx.yml

# 4. Configure output in filebeat.yml
sudo vi /etc/filebeat/filebeat.yml
# Set output.elasticsearch.hosts, username, password

# 5. Test configuration
sudo filebeat test config -e
sudo filebeat test output

# 6. Load dashboards, index templates, and ingest pipelines
sudo filebeat setup

# 7. Start Filebeat
sudo systemctl start filebeat
sudo systemctl enable filebeat

# 8. Verify data is flowing
curl -s -u elastic:changeme \
  "http://localhost:9200/filebeat-*/_count" | python3 -m json.tool

Expected output after a few seconds:

{
  "count": 1542,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}

Quick Reference

Task Command
List modules filebeat modules list
Enable module filebeat modules enable <name>
Disable module filebeat modules disable <name>
Module configs /etc/filebeat/modules.d/
Load dashboards filebeat setup --dashboards
Load pipelines filebeat setup --pipelines
Load index mgmt filebeat setup --index-management
Full setup filebeat setup
Override paths var.paths: ["/custom/path/*.log"]

Next: Metricbeat and Heartbeat — collecting system metrics and monitoring uptime.

You may also enjoy