ELK Stack — Install Filebeat

Install Filebeat

Filebeat is a lightweight log shipper that runs on your servers, tails log files, and sends events to Logstash or Elasticsearch. It uses minimal resources — perfect for running on production servers.

Install on Ubuntu/Debian

$ sudo apt install filebeat -y

Install on CentOS/RHEL

$ sudo yum install filebeat -y

Install on macOS (Homebrew)

$ brew install elastic/tap/filebeat-full

Verify Installation

$ filebeat version

Expected output:

filebeat version 8.15.0 (amd64), libbeat 8.15.0

Configure Filebeat

Edit the Filebeat configuration:

$ sudo nano /etc/filebeat/filebeat.yml

Minimal Configuration — Ship to Elasticsearch Directly

# ============================== Filebeat inputs ===============================
filebeat.inputs:
  - type: filestream
    id: system-logs
    enabled: true
    paths:
      - /var/log/syslog
      - /var/log/auth.log

# ============================== Elasticsearch Output ==========================
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "elastic"
  password: "YOUR_PASSWORD"
  ssl.verification_mode: none

# ============================== Processors ====================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~

Alternative — Ship to Logstash

If you want Logstash to process logs before they reach Elasticsearch:

# ============================== Filebeat inputs ===============================
filebeat.inputs:
  - type: filestream
    id: nginx-access
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      log_type: nginx
    fields_under_root: true

  - type: filestream
    id: app-logs
    enabled: true
    paths:
      - /var/log/myapp/*.log
    fields:
      log_type: application
    fields_under_root: true

# ============================== Logstash Output ===============================
output.logstash:
  hosts: ["localhost:5044"]

Important: You can only have ONE output configured at a time (either output.elasticsearch or output.logstash, not both).

Enable Filebeat Modules

Filebeat comes with built-in modules for common log formats. These modules know how to parse NGINX, Apache, MySQL, and many other log formats automatically.

List available modules:

$ sudo filebeat modules list

Expected output:

Enabled:

Disabled:
apache
auditd
elasticsearch
haproxy
kafka
kibana
logstash
mysql
nginx
osquery
postgresql
redis
system
...

Enable the system module:

$ sudo filebeat modules enable system

Expected output:

Enabled system

Enable the NGINX module:

$ sudo filebeat modules enable nginx

Configure a module:

$ sudo nano /etc/filebeat/modules.d/system.yml

- module: system
  syslog:
    enabled: true
  auth:
    enabled: true

Set Up Index Templates and Dashboards

Filebeat can automatically set up index templates and pre-built Kibana dashboards:

$ sudo filebeat setup -e \
  -E output.elasticsearch.hosts=["https://localhost:9200"] \
  -E output.elasticsearch.username=elastic \
  -E output.elasticsearch.password=YOUR_PASSWORD \
  -E output.elasticsearch.ssl.verification_mode=none

Expected output (truncated):

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded Ingest pipelines

This creates:

• Index template — defines field mappings for Filebeat data
• ILM policy — manages index lifecycle (rollover, deletion)
• Kibana dashboards — pre-built visualizations for enabled modules

Start Filebeat

$ sudo systemctl daemon-reload
$ sudo systemctl enable filebeat
$ sudo systemctl start filebeat

Verify Filebeat Is Working

Check the service:

$ sudo systemctl status filebeat

Check Filebeat logs:

$ sudo tail -f /var/log/filebeat/filebeat

Look for:

INFO    [publisher_pipeline_output] Connection to backoff(elasticsearch(https://localhost:9200)) established

Verify data in Elasticsearch:

$ curl -k -u elastic:<PASSWORD> "https://localhost:9200/_cat/indices?v&s=index" | grep filebeat

Expected output:

green  open filebeat-8.15.0-2025.03.15-000001 abc123 1 0 5432 0 2.1mb 2.1mb

Check in Kibana:

1. Open Kibana → Discover
2. Create a data view for filebeat-*
3. You should see log events flowing in

Test with a Custom Log File

Create a test log file:

$ echo '{"message": "Test log from Filebeat", "level": "info"}' >> /tmp/test.log

Configure Filebeat to read it:

filebeat.inputs:
  - type: filestream
    id: test-logs
    enabled: true
    paths:
      - /tmp/test.log

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "elastic"
  password: "YOUR_PASSWORD"
  ssl.verification_mode: none

Restart Filebeat:

$ sudo systemctl restart filebeat

Add more test lines:

$ for i in $(seq 1 10); do echo "{\"message\": \"Log line $i\", \"level\": \"info\"}" >> /tmp/test.log; done

Check in Kibana Discover — you should see 11 new events.

Important File Locations

File	Path
Configuration	/etc/filebeat/filebeat.yml
Modules	/etc/filebeat/modules.d/
Data directory	/var/lib/filebeat
Log directory	/var/log/filebeat
Binaries	/usr/share/filebeat/bin/
Registry	/var/lib/filebeat/registry/

Common Issues

Issue	Solution
No data in Elasticsearch	Check output.elasticsearch config; verify password
“Harvester could not be started”	Check file permissions — Filebeat needs read access
Duplicate events after restart	Filebeat tracks position in the registry; delete /var/lib/filebeat/registry/ to re-read
Cannot connect to Logstash	Verify Logstash is running with beats input on port 5044
Module not parsing correctly	Run sudo filebeat setup to install ingest pipelines

Quick Reference

Task	Command
Start Filebeat	sudo systemctl start filebeat
Stop Filebeat	sudo systemctl stop filebeat
Check status	sudo systemctl status filebeat
View logs	sudo tail -f /var/log/filebeat/filebeat
List modules	sudo filebeat modules list
Enable module	sudo filebeat modules enable <module>
Setup dashboards	sudo filebeat setup -e
Test config	sudo filebeat test config
Test output	sudo filebeat test output

Next Step

All individual components are installed! Let’s now set up the entire stack with Docker Compose — the fastest way to get a complete ELK environment running.