ELK Stack — Practice Projects: Exercises & Challenges

10 minute read

Practice Exercises & Challenges

These five challenge exercises extend the projects you built in this chapter. Each exercise builds on an existing project and pushes you to apply what you have learned in a new way.

Try to solve each challenge on your own before looking at the solution.

Exercise 1 — Add GeoIP Enrichment to the Nginx Project

Based on: Project 1 — Nginx Log Monitoring

Challenge

The Nginx project’s Logstash pipeline already includes a geoip filter, but all the traffic comes from localhost / Docker internal IPs, so GeoIP never resolves to a real location.

Your task:

  1. Modify the Nginx log format to include the X-Forwarded-For header
  2. Create a simple script that sends requests with a spoofed X-Forwarded-For header containing real-world public IPs
  3. Update the Logstash pipeline to extract the IP from X-Forwarded-For and apply GeoIP to it
  4. Build a Coordinate Map in Kibana showing request origins around the world

Hints

  • Use curl -H "X-Forwarded-For: 8.8.8.8" to simulate external IPs
  • In the Nginx log format add $http_x_forwarded_for
  • In Logstash, check if the field exists and prefer it over client_ip
  • The GeoIP database bundled with Logstash covers most public IPs
Show Solution

Step 1 — Update Nginx log format

log_format main '$remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent" "$http_x_forwarded_for"';

Step 2 — Create traffic generator script

#!/bin/bash
# generate-geo-traffic.sh

PUBLIC_IPS=(
  "8.8.8.8"          # US
  "1.1.1.1"          # Australia
  "103.21.244.0"     # India
  "185.70.41.35"     # Norway
  "41.57.156.0"      # Nigeria
  "189.45.67.1"      # Brazil
  "109.232.227.132"  # Germany
  "202.14.120.1"     # Japan
  "5.135.199.0"      # France
  "46.165.220.0"     # Russia
)

for i in $(seq 1 200); do
  IP=${PUBLIC_IPS[$((RANDOM % ${#PUBLIC_IPS[@]}))]}
  curl -s -H "X-Forwarded-For: $IP" http://localhost:8080/ > /dev/null
  sleep 0.1
done

echo "Generated 200 geo-distributed requests"

Step 3 — Update Logstash grok to capture forwarded IP

Update the grok pattern to include the forwarded-for field:

grok {
  match => {
    "message" => '%{IPORHOST:client_ip} - %{DATA:remote_user} \[%{HTTPDATE:timestamp}\] "%{WORD:http_method} %{URIPATHPARAM:request_uri} HTTP/%{NUMBER:http_version}" %{NUMBER:response_code:integer} %{NUMBER:bytes_sent:integer} "%{DATA:referrer}" "%{DATA:user_agent}" "%{DATA:x_forwarded_for}"'
  }
}

# Prefer X-Forwarded-For for GeoIP
if [x_forwarded_for] and [x_forwarded_for] != "-" {
  mutate {
    copy => { "x_forwarded_for" => "geo_source_ip" }
  }
} else {
  mutate {
    copy => { "client_ip" => "geo_source_ip" }
  }
}

geoip {
  source => "geo_source_ip"
  target => "geoip"
}

Step 4 — Kibana Map

  1. Create a data view for nginx-access-*
  2. Add a Maps visualization
  3. Add layer → Documents → select nginx-access-*
  4. Set the Geo field to geoip.location
  5. Use the response_code field for color coding

Exercise 2 — Implement Log-Based Alerting for Microservices

Based on: Project 2 — Microservices Logging

Challenge

The microservices project collects logs but does not alert when things go wrong. Your task:

  1. Create an Elasticsearch Watcher (or Kibana alerting rule) that fires when the payment failure rate exceeds 20% in a 5-minute window
  2. Create a second alert that fires when any service produces more than 50 ERROR logs in 10 minutes
  3. Configure the alert to write to a dedicated alerts-* index (simulating a notification)
  4. Build a Kibana panel that shows alert history over time

Hints

  • Use the _search API with aggregations to prototype the query
  • Kibana Rules → Create Rule → Elasticsearch query
  • For the payment failure rate, aggregate by log_message and calculate the ratio
  • You can also use Logstash to detect error spikes with the throttle filter
Show Solution

Alert 1 — Payment failure rate

Kibana Rule configuration:

  • Type: Elasticsearch query
  • Index: microservices-payment-service-*
  • Query:
{
  "bool": {
    "must": [
      { "match": { "log_message": "Payment failed" } },
      { "range": { "@timestamp": { "gte": "now-5m" } } }
    ]
  }
}
  • Threshold: Count > 10 (assuming ~50 total payments in 5 min = 20% failure)
  • Check every: 1 minute
  • Action: Index connector → write to alerts-payment-failures

Alert 2 — Error spike detection

  • Type: Elasticsearch query
  • Index: microservices-*
  • Query:
{
  "bool": {
    "must": [
      { "term": { "log_level": "ERROR" } },
      { "range": { "@timestamp": { "gte": "now-10m" } } }
    ]
  }
}
  • Threshold: Count > 50
  • Group by: service_name
  • Action: Index connector → write to alerts-error-spikes

Alert 3 — Logstash throttle approach (alternative)

Add to the microservices Logstash pipeline:

if [log_level] == "ERROR" {
  throttle {
    before_count => 50
    after_count => 51
    period => 600
    key => "%{service_name}"
    add_tag => ["error_spike_alert"]
  }
}

# In output section:
if "error_spike_alert" in [tags] {
  elasticsearch {
    hosts => ["http://es01:9200"]
    index => "alerts-error-spikes-%{+YYYY.MM.dd}"
  }
}

Kibana panel

Create a data view for alerts-* and add a data table panel sorted by @timestamp descending.

Exercise 3 — Scale ECK Deployment with Hot-Warm Architecture

Based on: Project 3 — ELK on Kubernetes with ECK

Challenge

The ECK project uses a single node set. In production, you need a hot-warm-cold architecture to manage costs and performance. Your task:

  1. Modify the Elasticsearch manifest to have 3 hot nodes and 2 warm nodes
  2. Create an Index Lifecycle Management (ILM) policy that moves indices from hot to warm after 7 days and deletes after 30 days
  3. Create an index template that applies the ILM policy to all k8s-* indices
  4. Verify that new indices are created on hot nodes

Hints

  • Use node.roles: ["data_hot", "ingest"] and node.roles: ["data_warm"]
  • Warm nodes can have smaller CPU/memory but larger storage
  • Use the Elasticsearch ILM API to create the policy
  • Use _cat/shards to verify shard placement
Show Solution

Step 1 — Updated Elasticsearch manifest

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elk-project
  namespace: elk-project
spec:
  version: 8.13.0
  nodeSets:
    - name: hot
      count: 3
      config:
        node.roles: ["master", "data_hot", "ingest"]
        node.store.allow_mmap: false
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 2Gi
                  cpu: "1"
                limits:
                  memory: 2Gi
                  cpu: "1"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 20Gi
            storageClassName: fast-ssd  # adjust for your cluster

    - name: warm
      count: 2
      config:
        node.roles: ["data_warm"]
        node.store.allow_mmap: false
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 1Gi
                  cpu: 500m
                limits:
                  memory: 1Gi
                  cpu: 500m
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 50Gi
            storageClassName: standard-hdd  # adjust for your cluster

Step 2 — ILM Policy

curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/_ilm/policy/k8s-logs-policy" \
  -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "10gb"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "require": { "data": "warm" }
          },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

Step 3 — Index Template

curl -sk -u "elastic:$ES_PASSWORD" -X PUT "https://localhost:9200/_index_template/k8s-logs-template" \
  -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["k8s-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "k8s-logs-policy",
      "index.routing.allocation.require.data": "hot",
      "number_of_replicas": 1
    }
  },
  "priority": 200
}'

Step 4 — Verify shard placement

curl -sk -u "elastic:$ES_PASSWORD" "https://localhost:9200/_cat/shards/k8s-*?v&h=index,shard,prirep,state,node"

All primary shards for new indices should be on nodes whose names contain hot.

Exercise 4 — Add Custom Dashboards to the Multi-Source Pipeline

Based on: Project 4 — Multi-Source Log Aggregation

Challenge

The multi-source project routes logs to separate indices but does not have a unified operations dashboard. Your task:

  1. Create a cross-index dashboard in Kibana that shows a unified view of all three log sources
  2. Add a saved search that lets operators filter by source type, severity, and time range
  3. Create a Vega visualization (or Lens) that shows the correlation between Apache error spikes and application errors on a dual-axis chart
  4. Export the dashboard as an NDJSON file so it can be version-controlled

Hints

  • Use a data view with pattern apache-*,syslog-*,app-*
  • Use the _index field to distinguish sources in aggregations
  • For the dual-axis chart, use Lens with a breakdown by index pattern
  • Export via Stack Management → Saved Objects → Export
Show Solution

Step 1 — Cross-index data view

  1. Go to Stack Management → Data Views
  2. Create: name all-sources, pattern apache-*,syslog-*,app-logs-*
  3. Time field: @timestamp

Step 2 — Saved search

  1. Go to Discover
  2. Select the all-sources data view
  3. Add filters:
    • Add column: _index (to see the source)
    • Add column: severity or response_category or level
  4. Save as “All Sources — Unified View”

Step 3 — Correlation chart using Lens

  1. Create a new visualization → Lens
  2. Data view: all-sources
  3. X-axis: @timestamp with date histogram
  4. Left Y-axis: Count of documents where _index starts with apache-error
  5. Right Y-axis: Count of documents where _index starts with app-logs AND level: ERROR
  6. Layer 1: Apache errors (bar chart)
  7. Layer 2: App errors (line chart on right axis)
  8. Save as “Error Correlation — Apache vs App”

Step 4 — Export dashboard

# Via Kibana API
curl -s http://localhost:5601/api/saved_objects/_export \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -d '{"type": ["dashboard"], "includeReferencesDeep": true}' \
  -o dashboard-export.ndjson

echo "Dashboard exported to dashboard-export.ndjson"

Or via the UI:

  1. Go to Stack Management → Saved Objects
  2. Select the dashboard
  3. Click Export → include related objects
  4. Save the .ndjson file in your project repository

Exercise 5 — Build Automated Threat Response

Based on: Project 5 — Security Dashboard

Challenge

The security project detects threats but does not respond to them. Your task:

  1. Create a Logstash pipeline that monitors the security-alerts-* index for brute force IPs (more than 10 failed logins in 5 minutes)
  2. When a brute force is detected, write the offending IP to a file (/tmp/blocked-ips.txt)
  3. Create a script that reads this file and could theoretically add iptables rules (simulate with a log message)
  4. Build a Kibana panel showing the timeline of automated blocks
  5. Bonus: Use the Elasticsearch _update_by_query to tag all historical events from a blocked IP

Hints

  • Use a second Logstash pipeline that reads from Elasticsearch using the elasticsearch input
  • Use the file output plugin to write blocked IPs
  • Use the exec output plugin to run a script
  • For the bonus, use curl with _update_by_query
Show Solution

Step 1 — Threat response Logstash pipeline

Add to pipelines.yml:

- pipeline.id: threat-response
  path.config: "/usr/share/logstash/pipeline-threat/response.conf"
  pipeline.workers: 1

Create logstash/pipeline-threat/response.conf:

input {
  elasticsearch {
    hosts => ["http://es01:9200"]
    index => "security-auth-*"
    query => '{"query":{"bool":{"must":[{"term":{"security_event":"failed_authentication"}},{"range":{"@timestamp":{"gte":"now-5m"}}}]}}}'
    schedule => "*/1 * * * *"
    size => 1000
  }
}

filter {
  # Count failures per source IP
  aggregate {
    task_id => "%{source_ip}"
    code => "
      map['attempts'] ||= 0
      map['attempts'] += 1
      map['source_ip'] = event.get('source_ip')
      event.cancel()
    "
    push_map_as_event_on_timeout => true
    timeout => 10
    timeout_tags => ['aggregated']
  }

  if "aggregated" in [tags] {
    if [attempts] and [attempts] >= 10 {
      mutate {
        add_tag => ["auto_block"]
        add_field => {
          "action" => "block"
          "reason" => "brute_force_detected"
          "blocked_at" => "%{@timestamp}"
        }
      }
    } else {
      drop {}
    }
  }
}

output {
  if "auto_block" in [tags] {
    # Write blocked IP to file
    file {
      path => "/tmp/blocked-ips.txt"
      codec => line { format => "%{source_ip}" }
    }

    # Log the block action to Elasticsearch
    elasticsearch {
      hosts => ["http://es01:9200"]
      index => "security-auto-blocks-%{+YYYY.MM.dd}"
    }

    # Simulate iptables block (use exec for real scenarios)
    stdout {
      codec => line { format => "AUTO-BLOCK: Blocking IP %{source_ip} — %{attempts} failed attempts" }
    }
  }
}

Step 2 — Block script (simulated)

#!/bin/bash
# simulate-block.sh — reads blocked IPs and simulates iptables rules

BLOCKED_FILE="/tmp/blocked-ips.txt"

if [ ! -f "$BLOCKED_FILE" ]; then
  echo "No blocked IPs found"
  exit 0
fi

while IFS= read -r ip; do
  echo "[$(date)] SIMULATED: iptables -A INPUT -s $ip -j DROP"
  # In production: sudo iptables -A INPUT -s "$ip" -j DROP
done < "$BLOCKED_FILE"

Step 3 — Kibana panel

Create a data view for security-auto-blocks-* and add:

  • Timeline: Line chart of blocks over time
  • Blocked IPs table: Data table with source_ip, attempts, blocked_at
  • Map: GeoIP locations of blocked IPs

Step 4 (Bonus) — Tag historical events from blocked IP

# Replace BLOCKED_IP with the actual IP
BLOCKED_IP="185.220.101.1"

curl -s -X POST "http://localhost:9200/security-*/_update_by_query" \
  -H 'Content-Type: application/json' -d"
{
  \"query\": {
    \"term\": { \"source_ip\": \"$BLOCKED_IP\" }
  },
  \"script\": {
    \"source\": \"ctx._source.tags = ctx._source.tags != null ? ctx._source.tags : []; ctx._source.tags.add('retroactively_blocked')\",
    \"lang\": \"painless\"
  }
}"

This tags all historical events from the blocked IP so analysts can review the full activity history.

Summary

Exercise Skill Practiced
1 — GeoIP Enrichment Log format modification, GeoIP filtering, map visualizations
2 — Alerting Kibana rules, Elasticsearch queries, threshold-based alerting
3 — Hot-Warm Architecture ILM policies, node roles, index templates, capacity planning
4 — Custom Dashboards Cross-index views, Lens visualizations, saved object export
5 — Automated Response Pipeline chaining, aggregate filter, automated threat mitigation

What’s Next

Congratulations on completing the Practice Projects chapter! You now have hands-on experience with:

  • End-to-end ELK pipelines for web servers, microservices, and Kubernetes
  • Multi-pipeline Logstash with custom grok patterns
  • Security dashboards with threat enrichment and alerting
  • Advanced topics like ILM, hot-warm architecture, and automated response

Continue exploring by modifying these projects for your own infrastructure and log sources.

→ Back to Practice Projects Hub

You may also enjoy