ELK Stack — Beats Exercises

8 minute read

Practice Exercises

These exercises reinforce the Beats concepts covered in this chapter. Each exercise increases in difficulty from basic setup to advanced processor configurations.

Exercise 1: Basic Filebeat Setup to Ship Application Logs

Scenario: You have a Java application that writes logs to /var/log/webapp/application.log. The logs follow a standard pattern with timestamps. You need to ship these logs to your Elasticsearch cluster at https://es-node-1:9200.

Requirements:

  1. Configure Filebeat to monitor /var/log/webapp/application.log
  2. Add custom fields: app: webapp, environment: staging
  3. Place custom fields at the root level of the event
  4. Tag events with webapp and staging
  5. Exclude lines starting with TRACE
  6. Configure output to Elasticsearch with basic authentication
  7. Enable JSON logging to /var/log/filebeat/
Show Solution 
# /etc/filebeat/filebeat.yml

filebeat.inputs:
  - type: log
    id: webapp-logs
    enabled: true
    paths:
      - /var/log/webapp/application.log
    fields:
      app: webapp
      environment: staging
    fields_under_root: true
    tags: ["webapp", "staging"]
    exclude_lines: ['^TRACE']

output.elasticsearch:
  hosts: ["https://es-node-1:9200"]
  username: "filebeat_writer"
  password: "changeme"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/filebeat/certs/ca.crt"]

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Test and start:

# Test configuration
sudo filebeat test config -e

# Test output connectivity
sudo filebeat test output

# Start Filebeat
sudo systemctl start filebeat
sudo systemctl enable filebeat

# Verify data is flowing
curl -s -u elastic:changeme \
  "https://es-node-1:9200/filebeat-*/_search?size=1" | python3 -m json.tool

Exercise 2: Enabling and Configuring Filebeat Modules

Scenario: You manage a web server stack running Nginx and MySQL. You want Filebeat to collect logs from both services and set up the pre-built Kibana dashboards.

Requirements:

  1. Enable the system, nginx, and mysql modules
  2. Configure the Nginx module to read access logs from /opt/nginx/logs/access.log and error logs from /opt/nginx/logs/error.log
  3. Configure the MySQL module to read error logs from /var/log/mysql/error.log and slow query logs from /var/log/mysql/slow-query.log
  4. Keep system module defaults
  5. Load dashboards into Kibana
  6. Verify data is indexed
Show Solution

Step 1: Enable the modules.

sudo filebeat modules enable system nginx mysql

Step 2: Configure the Nginx module.

# /etc/filebeat/modules.d/nginx.yml
- module: nginx
  access:
    enabled: true
    var.paths: ["/opt/nginx/logs/access.log*"]
  error:
    enabled: true
    var.paths: ["/opt/nginx/logs/error.log*"]
  ingress_controller:
    enabled: false

Step 3: Configure the MySQL module.

# /etc/filebeat/modules.d/mysql.yml
- module: mysql
  error:
    enabled: true
    var.paths: ["/var/log/mysql/error.log*"]
  slowlog:
    enabled: true
    var.paths: ["/var/log/mysql/slow-query.log*"]

Step 4: Keep system module at defaults.

# /etc/filebeat/modules.d/system.yml
- module: system
  syslog:
    enabled: true
  auth:
    enabled: true

Step 5: Load dashboards, index templates, and ingest pipelines.

# Full setup — loads dashboards, index templates, ILM policies, and ingest pipelines
sudo filebeat setup

# Or load only dashboards
sudo filebeat setup --dashboards

Step 6: Restart Filebeat and verify.

sudo systemctl restart filebeat

# Check Nginx index
curl -s -u elastic:changeme \
  "https://es-node-1:9200/filebeat-*/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 1,
    "query": {
      "match": {
        "event.module": "nginx"
      }
    }
  }' | python3 -m json.tool

# Check MySQL index
curl -s -u elastic:changeme \
  "https://es-node-1:9200/filebeat-*/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 1,
    "query": {
      "match": {
        "event.module": "mysql"
      }
    }
  }' | python3 -m json.tool

Open Kibana → Dashboard → search for [Filebeat Nginx] to see the pre-built visualizations.

Exercise 3: Metricbeat System Monitoring

Scenario: You need to monitor system health on a production database server. Collect CPU, memory, disk, network, filesystem, and process metrics. Track the top 5 processes by CPU and memory. Send metrics to Elasticsearch every 15 seconds.

Requirements:

  1. Configure Metricbeat with the system module
  2. Collect: cpu, memory, load, network, diskio, filesystem, process, process_summary
  3. Set collection period to 15 seconds
  4. Track top 5 processes by CPU and top 5 by memory
  5. Add host metadata and cloud metadata processors
  6. Add a custom field role: database-server
  7. Output to Elasticsearch with TLS
Show Solution 
# /etc/metricbeat/metricbeat.yml

metricbeat.modules:
  - module: system
    metricsets:
      - cpu
      - memory
      - load
      - network
      - diskio
      - filesystem
      - process
      - process_summary
    enabled: true
    period: 15s
    processes: ['.*']
    process.include_top_n:
      by_cpu: 5
      by_memory: 5
    cpu.metrics: ["percentages", "normalized_percentages"]
    core.metrics: ["percentages"]

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_fields:
      target: ""
      fields:
        role: database-server

output.elasticsearch:
  hosts: ["https://es-node-1:9200"]
  username: "metricbeat_writer"
  password: "changeme"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/metricbeat/certs/ca.crt"]
    certificate: "/etc/metricbeat/certs/metricbeat.crt"
    key: "/etc/metricbeat/certs/metricbeat.key"

setup.kibana:
  host: "https://kibana-host:5601"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/metricbeat/certs/ca.crt"]

setup.dashboards.enabled: true

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7

Test and start:

# Test configuration
sudo metricbeat test config -e

# Test output
sudo metricbeat test output

# Load dashboards
sudo metricbeat setup --dashboards

# Start Metricbeat
sudo systemctl start metricbeat
sudo systemctl enable metricbeat

# Verify metrics are flowing (check CPU metricset)
curl -s -u elastic:changeme \
  "https://es-node-1:9200/metricbeat-*/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 1,
    "query": {
      "match": {
        "metricset.name": "cpu"
      }
    },
    "sort": [{"@timestamp": "desc"}]
  }' | python3 -m json.tool

Open Kibana → Dashboard → [Metricbeat System] Host overview to see CPU, memory, and disk graphs.

Exercise 4: Adding Processors to Filebeat

Scenario: Your application writes JSON-formatted logs. You need to parse the JSON, add infrastructure metadata, remove sensitive fields, drop health check noise, and add custom tags for routing.

Sample log line:

{"timestamp":"2026-04-01T14:49:00Z","level":"info","service":"order-api","user_id":"u-12345","auth_token":"Bearer eyJhbGci...","message":"Order created","trace_id":"t-abc-123"}

Requirements:

  1. Parse the JSON message field into top-level event fields
  2. Add host metadata and Docker metadata
  3. Drop the auth_token field (sensitive data)
  4. Drop events where message contains GET /healthz
  5. Rename level to log.level and service to service.name
  6. Add custom fields: environment: production, team: commerce
  7. Drop events with empty messages
Show Solution 
# /etc/filebeat/filebeat.yml

filebeat.inputs:
  - type: log
    id: order-api-logs
    enabled: true
    paths:
      - /var/log/order-api/*.log
    tags: ["order-api", "json"]

processors:
  # 1. Parse JSON log lines into structured fields
  - decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true
      expand_keys: true
      add_error_key: true

  # 2. Add infrastructure metadata
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"
      match_source: true

  # 3. Drop sensitive fields
  - drop_fields:
      fields:
        - "auth_token"
        - "agent.ephemeral_id"
      ignore_missing: true

  # 4. Drop health check noise
  - drop_event:
      when:
        contains:
          message: "GET /healthz"

  # 5. Drop events with empty messages
  - drop_event:
      when:
        regexp:
          message: "^\\s*$"

  # 6. Rename fields for ECS compliance
  - rename:
      fields:
        - from: "level"
          to: "log.level"
        - from: "service"
          to: "service.name"
      ignore_missing: true
      fail_on_error: false

  # 7. Add custom fields for routing
  - add_fields:
      target: ""
      fields:
        environment: production
        team: commerce

output.elasticsearch:
  hosts: ["https://es-node-1:9200"]
  username: "filebeat_writer"
  password: "changeme"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/filebeat/certs/ca.crt"]

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7

Test and verify:

# Test config
sudo filebeat test config -e

# Start in debug mode to see processor output
sudo filebeat -e -d "processors"

# After starting, check that auth_token is NOT in indexed events
curl -s -u elastic:changeme \
  "https://es-node-1:9200/filebeat-*/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 1,
    "query": {
      "exists": {
        "field": "auth_token"
      }
    }
  }' | python3 -m json.tool

Expected: zero hits (auth_token was stripped by drop_fields).

The indexed event should look like:

{
  "@timestamp": "2026-04-01T14:49:00.000Z",
  "log.level": "info",
  "service.name": "order-api",
  "user_id": "u-12345",
  "message": "Order created",
  "trace_id": "t-abc-123",
  "environment": "production",
  "team": "commerce",
  "host": {
    "name": "web-01",
    "hostname": "web-01",
    "architecture": "x86_64",
    "os": { "platform": "ubuntu", "version": "22.04" }
  },
  "tags": ["order-api", "json"]
}

Exercise 5: Heartbeat Uptime Monitoring

Scenario: You need to set up uptime monitoring for your production infrastructure. Monitor a web application (HTTPS), a REST API health endpoint, a PostgreSQL database (TCP), a Redis cache (TCP), and three internal servers (ICMP ping).

Requirements:

  1. HTTP monitor: check https://app.example.com every 30 seconds — verify status 200 and body contains “ok”
  2. HTTP monitor: check https://api.example.com/health every 60 seconds — verify JSON response contains "status": "healthy"
  3. TCP monitor: check PostgreSQL on db-primary:5432 and db-replica:5432 every 30 seconds
  4. TCP monitor: check Redis on redis-01:6379 and redis-02:6379 every 15 seconds
  5. ICMP monitor: ping 10.0.1.10, 10.0.1.11, 10.0.1.12 every 60 seconds
  6. Add appropriate tags to each monitor
  7. Load Heartbeat dashboards into Kibana
Show Solution 
# /etc/heartbeat/heartbeat.yml

heartbeat.monitors:

  # 1. Web application — HTTPS with body check
  - type: http
    id: web-app
    name: "Production Web App"
    enabled: true
    urls:
      - "https://app.example.com"
    schedule: '@every 30s'
    timeout: 10s
    check.response:
      status: [200]
      body:
        - "ok"
    ssl:
      verification_mode: full
    tags: ["web", "production", "critical"]

  # 2. API health endpoint — HTTPS with JSON check
  - type: http
    id: api-health
    name: "REST API Health"
    enabled: true
    urls:
      - "https://api.example.com/health"
    schedule: '@every 60s'
    timeout: 10s
    check.response:
      status: [200]
      json:
        - description: "API reports healthy"
          condition:
            contains:
              status: "healthy"
    ssl:
      verification_mode: full
    tags: ["api", "production", "critical"]

  # 3. PostgreSQL — TCP port check
  - type: tcp
    id: postgres-check
    name: "PostgreSQL"
    enabled: true
    hosts:
      - "db-primary:5432"
      - "db-replica:5432"
    schedule: '@every 30s'
    timeout: 5s
    tags: ["database", "postgresql", "production"]

  # 4. Redis — TCP port check
  - type: tcp
    id: redis-check
    name: "Redis Cache"
    enabled: true
    hosts:
      - "redis-01:6379"
      - "redis-02:6379"
    schedule: '@every 15s'
    timeout: 3s
    tags: ["cache", "redis", "production"]

  # 5. Server ping — ICMP check
  - type: icmp
    id: server-ping
    name: "Internal Servers"
    enabled: true
    hosts:
      - "10.0.1.10"
      - "10.0.1.11"
      - "10.0.1.12"
    schedule: '@every 60s'
    timeout: 5s
    wait: 1s
    tags: ["infrastructure", "ping"]

# Processors
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

# Output
output.elasticsearch:
  hosts: ["https://es-node-1:9200"]
  username: "heartbeat_writer"
  password: "changeme"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/heartbeat/certs/ca.crt"]

# Kibana
setup.kibana:
  host: "https://kibana-host:5601"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/heartbeat/certs/ca.crt"]

setup.dashboards.enabled: true

# Logging
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/heartbeat
  name: heartbeat
  keepfiles: 7

Test, load dashboards, and start:

# Test config
sudo heartbeat test config -e

# Test output
sudo heartbeat test output

# Load dashboards
sudo heartbeat setup --dashboards

# Start Heartbeat
sudo systemctl start heartbeat-elastic
sudo systemctl enable heartbeat-elastic

# Verify uptime data is flowing
curl -s -u elastic:changeme \
  "https://es-node-1:9200/heartbeat-*/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 5,
    "sort": [{"@timestamp": "desc"}],
    "_source": ["monitor.name", "monitor.status", "url.full", "monitor.type"]
  }' | python3 -m json.tool

Expected output:

{
  "hits": {
    "hits": [
      {
        "_source": {
          "monitor.name": "Production Web App",
          "monitor.status": "up",
          "url.full": "https://app.example.com",
          "monitor.type": "http"
        }
      },
      {
        "_source": {
          "monitor.name": "PostgreSQL",
          "monitor.status": "up",
          "url.full": "tcp://db-primary:5432",
          "monitor.type": "tcp"
        }
      }
    ]
  }
}

Open Kibana → Observability → Uptime to see all monitors, their status, response times, and availability percentages.

Summary

Exercise Skill Practiced
1 Basic Filebeat input, output, and logging configuration
2 Filebeat modules — enabling, configuring, and loading dashboards
3 Metricbeat system module — metrics collection and process monitoring
4 Processors — JSON parsing, field manipulation, conditional filtering
5 Heartbeat — HTTP, TCP, and ICMP uptime monitoring

Return to Beats Hub to review chapter topics.

You may also enjoy