ELK Stack — Production Exercises

10 minute read

Practice Exercises — Production

These exercises cover the key production topics: high availability, backup & restore, monitoring, capacity planning, and upgrade strategies. Try each exercise on your own before expanding the solution.

Exercise 1: Design a Highly Available Cluster

Scenario: Your company needs a production Elasticsearch cluster deployed across 3 availability zones. Requirements:

  • Survive the loss of one entire AZ
  • Handle 200 GB/day ingest with 30-day hot retention and 90-day warm retention
  • Dedicated master nodes
  • Zone-aware shard allocation

Task: Write the complete elasticsearch.yml for a master node in AZ-1 and a hot data node in AZ-2. Include cluster name, node roles, discovery, and zone awareness settings.

Show Solution

Master Node — AZ-1 (es-master-1)

# ── Cluster ──
cluster.name: prod-elk-cluster
node.name: es-master-1

# ── Node Roles ──
node.roles: [master]

# ── Paths ──
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

# ── Network ──
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

# ── Discovery ──
discovery.seed_hosts:
  - es-master-1:9300
  - es-master-2:9300
  - es-master-3:9300

# ── Zone Awareness ──
node.attr.zone: az-1
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: az-1,az-2,az-3

# ── Security ──
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

Hot Data Node — AZ-2 (es-hot-2a)

# ── Cluster ──
cluster.name: prod-elk-cluster
node.name: es-hot-2a

# ── Node Roles ──
node.roles: [data_hot, ingest]

# ── Paths ──
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

# ── Network ──
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

# ── Discovery ──
discovery.seed_hosts:
  - es-master-1:9300
  - es-master-2:9300
  - es-master-3:9300

# ── Zone Awareness ──
node.attr.zone: az-2
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: az-1,az-2,az-3

# ── Recovery ──
cluster.routing.allocation.node_concurrent_recoveries: 4

# ── Security ──
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

Cluster summary:

  • 3 dedicated master nodes (one per AZ)
  • 6 hot data nodes (2 per AZ) — 16 vCPU, 64 GB RAM, 2 TB NVMe each
  • 6 warm data nodes (2 per AZ) — 8 vCPU, 64 GB RAM, 8 TB SAS each
  • 3 Kibana instances behind a load balancer
  • Zone-aware allocation ensures replicas are in different AZs

Exercise 2: Set Up Snapshot Repository + SLM Policy

Scenario: Configure an S3-based snapshot repository and create an SLM policy that:

  • Takes daily snapshots at 03:00 UTC
  • Retains snapshots for 30 days
  • Keeps a minimum of 7 and maximum of 45 snapshots
  • Includes all indices but excludes global state

Task: Write the curl commands to register the repository and create the SLM policy. Then show how to execute the policy manually and verify the snapshot.

Show Solution

Step 1 — Register S3 Repository

# Install plugin (on every node, then restart)
bin/elasticsearch-plugin install repository-s3

# Add credentials to keystore
bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key

# Register the repository
curl -X PUT "localhost:9200/_snapshot/prod-s3-backup" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "company-es-backups",
    "region": "us-east-1",
    "base_path": "production/snapshots",
    "compress": true,
    "server_side_encryption": true,
    "storage_class": "standard_ia",
    "max_snapshot_bytes_per_sec": "250mb",
    "max_restore_bytes_per_sec": "250mb"
  }
}'

Step 2 — Create SLM Policy

curl -X PUT "localhost:9200/_slm/policy/daily-backup" -H 'Content-Type: application/json' -d'
{
  "schedule": "0 0 3 * * ?",
  "name": "<daily-snap-{now/d}>",
  "repository": "prod-s3-backup",
  "config": {
    "indices": ["*"],
    "ignore_unavailable": true,
    "include_global_state": false
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 7,
    "max_count": 45
  }
}'

Step 3 — Execute Policy Manually

curl -X POST "localhost:9200/_slm/policy/daily-backup/_execute"

Step 4 — Verify Snapshot

# Check SLM status
curl -s "localhost:9200/_slm/policy/daily-backup" | jq '.daily-backup.last_success'

# List snapshots
curl -s "localhost:9200/_snapshot/prod-s3-backup/*?verbose=false" | jq '.snapshots[-1]'

# Check snapshot detail
curl -s "localhost:9200/_snapshot/prod-s3-backup/daily-snap-2026.04.01/_status" | jq '.snapshots[0].state'

Expected output: Snapshot state should be SUCCESS.

Exercise 3: Build a Cluster Health Monitoring Dashboard

Scenario: Your team needs a monitoring script that runs every 5 minutes (via cron) and reports cluster health, node status, and key metrics. If anything is unhealthy, it should send an alert.

Task: Write a Bash script that:

  1. Checks cluster health status
  2. Checks for unassigned shards
  3. Checks JVM heap usage across all nodes
  4. Checks for thread pool rejections
  5. Outputs a summary and alerts if any metric is unhealthy
Show Solution 
#!/bin/bash
# es-health-check.sh — Elasticsearch cluster health monitor
# Usage: Run via cron every 5 minutes
# */5 * * * * /opt/scripts/es-health-check.sh >> /var/log/es-health.log 2>&1

ES_HOST="localhost:9200"
ALERT_EMAIL="ops-team@company.com"
HEAP_THRESHOLD=85
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
ALERTS=()

echo "=== ES Health Check — $TIMESTAMP ==="

# 1. Cluster health
HEALTH=$(curl -s "$ES_HOST/_cluster/health")
STATUS=$(echo "$HEALTH" | jq -r '.status')
UNASSIGNED=$(echo "$HEALTH" | jq -r '.unassigned_shards')
PENDING=$(echo "$HEALTH" | jq -r '.number_of_pending_tasks')

echo "Cluster Status: $STATUS"
echo "Unassigned Shards: $UNASSIGNED"
echo "Pending Tasks: $PENDING"

if [ "$STATUS" != "green" ]; then
  ALERTS+=("Cluster status is $STATUS")
fi
if [ "$UNASSIGNED" -gt 0 ]; then
  ALERTS+=("$UNASSIGNED unassigned shards detected")
fi
if [ "$PENDING" -gt 10 ]; then
  ALERTS+=("$PENDING pending cluster tasks")
fi

# 2. JVM heap usage per node
echo ""
echo "--- JVM Heap Usage ---"
curl -s "$ES_HOST/_nodes/stats/jvm" | jq -r '
  .nodes | to_entries[] |
  "\(.value.name): \(.value.jvm.mem.heap_used_percent)%"
' | while read line; do
  echo "$line"
  PERCENT=$(echo "$line" | grep -oE '[0-9]+%' | tr -d '%')
  if [ "$PERCENT" -gt "$HEAP_THRESHOLD" ]; then
    NODE_NAME=$(echo "$line" | cut -d: -f1)
    ALERTS+=("High JVM heap on $NODE_NAME: ${PERCENT}%")
  fi
done

# 3. Thread pool rejections
echo ""
echo "--- Thread Pool Rejections ---"
REJECTIONS=$(curl -s "$ES_HOST/_cat/thread_pool?v&h=node_name,name,rejected" | awk '$3 > 0')
if [ -n "$REJECTIONS" ]; then
  echo "$REJECTIONS"
  ALERTS+=("Thread pool rejections detected")
else
  echo "No rejections"
fi

# 4. Disk usage
echo ""
echo "--- Disk Usage ---"
curl -s "$ES_HOST/_cat/allocation?v&h=node,disk.percent" | tail -n +2 | while read line; do
  echo "$line"
  DISK_PCT=$(echo "$line" | awk '{print $2}')
  if [ "$DISK_PCT" -gt 85 ]; then
    NODE=$(echo "$line" | awk '{print $1}')
    ALERTS+=("High disk usage on $NODE: ${DISK_PCT}%")
  fi
done

# 5. Send alerts
echo ""
if [ ${#ALERTS[@]} -gt 0 ]; then
  echo "⚠️  ALERTS:"
  ALERT_BODY=""
  for alert in "${ALERTS[@]}"; do
    echo "  - $alert"
    ALERT_BODY="$ALERT_BODY\n- $alert"
  done
  echo -e "ES Cluster Alert at $TIMESTAMP\n$ALERT_BODY" | \
    mail -s "⚠️ ES Cluster Health Alert" "$ALERT_EMAIL"
else
  echo "✅ All checks passed"
fi

echo "=== End ==="

Make it executable and schedule:

chmod +x /opt/scripts/es-health-check.sh

# Add to crontab
crontab -e
# */5 * * * * /opt/scripts/es-health-check.sh >> /var/log/es-health.log 2>&1

Exercise 4: Capacity Planning for 500 GB/Day Ingest

Scenario: Your organization plans to deploy ELK for centralized logging with:

  • 500 GB/day raw log ingest
  • Hot retention: 7 days
  • Warm retention: 30 days
  • Cold retention: 90 days (searchable snapshots)
  • 1 replica for all tiers
  • Must survive one AZ failure (3 AZs)

Task: Calculate:

  1. Total storage needed per tier
  2. Number of nodes per tier (with recommended hardware)
  3. Total RAM and heap allocation
  4. Number of master nodes and coordinating nodes
Show Solution

Storage Calculations

Overhead factor = 1.15

Hot tier (7 days):
  500 GB × 7 × (1 + 1) × 1.15 = 8,050 GB ≈ 8.1 TB

Warm tier (30 days):
  500 GB × 30 × (1 + 1) × 1.15 = 34,500 GB ≈ 34.5 TB

Cold tier (90 days — searchable snapshots):
  Stored in S3/GCS — local cache only
  Cache estimate: ~10% of data = 500 × 90 × 0.10 = 4,500 GB ≈ 4.5 TB local cache
  S3 storage: 500 × 90 × 1.15 = 51,750 GB ≈ 51.8 TB (no replicas — S3 handles durability)

Node Count per Tier

Hot tier:
  8.1 TB / 2 TB per node = 4.05 → 6 nodes (2 per AZ for zone awareness)
  Hardware: 16 vCPU, 64 GB RAM, 2 TB NVMe SSD

Warm tier:
  34.5 TB / 8 TB per node = 4.3 → 6 nodes (2 per AZ)
  Hardware: 8 vCPU, 64 GB RAM, 8 TB SAS SSD

Cold tier (frozen):
  4.5 TB cache / 2 TB per node = 2.25 → 3 nodes (1 per AZ)
  Hardware: 4 vCPU, 32 GB RAM, 2 TB SSD (cache only)

Master & Coordinating Nodes

Master nodes: 3 (one per AZ)
  Hardware: 4 vCPU, 16 GB RAM, 100 GB SSD

Coordinating nodes: 3 (one per AZ)
  Hardware: 8 vCPU, 32 GB RAM

RAM & Heap Summary

Tier Nodes RAM/Node Heap/Node Total RAM
Master 3 16 GB 8 GB 48 GB
Hot 6 64 GB 31 GB 384 GB
Warm 6 64 GB 31 GB 384 GB
Cold/Frozen 3 32 GB 16 GB 96 GB
Coordinating 3 32 GB 16 GB 96 GB
Total 21     1,008 GB

Additional Components

Kibana: 3 instances (one per AZ) — 4 vCPU, 8 GB RAM
Logstash: 6 instances (2 per AZ) — 8 vCPU, 16 GB RAM
  (500 GB/day ÷ 6 ≈ 83 GB/day per Logstash instance)

Cost Estimate (AWS, on-demand)

Component Instance Type Count Monthly Cost (est.)
Master m6i.xlarge 3 ~$430
Hot data r6id.2xlarge 6 ~$2,600
Warm data d3.2xlarge 6 ~$3,000
Frozen m6i.xlarge 3 ~$430
Coordinating m6i.2xlarge 3 ~$830
Kibana m6i.xlarge 3 ~$430
Logstash c6i.2xlarge 6 ~$1,470
S3 (cold) 51.8 TB — ~$1,200
Total   30 ~$10,390/mo

Reserved instances or Savings Plans can reduce costs by 30–50%.

Exercise 5: Perform a Rolling Upgrade (8.10 → 8.12)

Scenario: Your 9-node cluster (3 masters, 6 data nodes) is running ES 8.10. You need to upgrade to 8.12 with zero downtime.

Task: Write the complete step-by-step commands for a rolling upgrade. Include pre-upgrade checks, the upgrade of one master node, and the upgrade of one data node.

Show Solution

Pre-Upgrade Steps

# 1. Check for deprecations
curl -s "localhost:9200/_migration/deprecations" | jq '.cluster_settings, .index_settings'

# 2. Ensure cluster is green
curl -s "localhost:9200/_cluster/health" | jq '{status, unassigned_shards}'

# 3. Take a full snapshot
curl -X PUT "localhost:9200/_snapshot/prod-s3-backup/pre-upgrade-8.12-$(date +%Y%m%d)?wait_for_completion=true" \
  -H 'Content-Type: application/json' -d'
{
  "indices": "*",
  "include_global_state": true
}'

# 4. Verify snapshot
curl -s "localhost:9200/_snapshot/prod-s3-backup/pre-upgrade-8.12-*" | jq '.snapshots[0].state'
# Must return "SUCCESS"

# 5. Check current versions
curl -s "localhost:9200/_cat/nodes?v&h=name,version,role"

Upgrade Master Node (es-master-1)

# Step 1 — Disable shard allocation
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

# Step 2 — Flush
curl -X POST "localhost:9200/_flush"

# Step 3 — Stop Elasticsearch on es-master-1
ssh es-master-1 "sudo systemctl stop elasticsearch"

# Step 4 — Upgrade package on es-master-1
ssh es-master-1 "sudo apt-get update && sudo apt-get install -y elasticsearch=8.12.0"

# Step 5 — Start Elasticsearch on es-master-1
ssh es-master-1 "sudo systemctl start elasticsearch"

# Step 6 — Wait for the node to rejoin
sleep 30
curl -s "localhost:9200/_cat/nodes?v&h=name,version" | grep master-1

# Step 7 — Re-enable shard allocation
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}'

# Step 8 — Wait for green
curl -s "localhost:9200/_cluster/health?wait_for_status=green&timeout=120s" | jq '{status}'

Repeat for es-master-2 and es-master-3.

Upgrade Data Node (es-data-1a)

# Step 1 — Disable shard allocation
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

# Step 2 — Flush
curl -X POST "localhost:9200/_flush"

# Step 3 — Stop Elasticsearch on es-data-1a
ssh es-data-1a "sudo systemctl stop elasticsearch"

# Step 4 — Upgrade package
ssh es-data-1a "sudo apt-get update && sudo apt-get install -y elasticsearch=8.12.0"

# Step 5 — Start Elasticsearch
ssh es-data-1a "sudo systemctl start elasticsearch"

# Step 6 — Wait for rejoin
sleep 30
curl -s "localhost:9200/_cat/nodes?v&h=name,version" | grep data-1a

# Step 7 — Re-enable allocation
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}'

# Step 8 — Wait for green (shard recovery takes longer for data nodes)
curl -s "localhost:9200/_cluster/health?wait_for_status=green&timeout=300s" | jq '{status, relocating_shards, initializing_shards}'

Repeat for es-data-1b, es-data-2a, es-data-2b, es-data-3a, es-data-3b.

Post-Upgrade — Upgrade Kibana & Beats

# Upgrade Kibana (all instances)
ssh kibana-1 "sudo systemctl stop kibana && sudo apt-get install -y kibana=8.12.0 && sudo systemctl start kibana"
ssh kibana-2 "sudo systemctl stop kibana && sudo apt-get install -y kibana=8.12.0 && sudo systemctl start kibana"
ssh kibana-3 "sudo systemctl stop kibana && sudo apt-get install -y kibana=8.12.0 && sudo systemctl start kibana"

# Verify Kibana version
curl -s "localhost:5601/api/status" | jq '.version.number'

# Upgrade Filebeat on all servers
ansible all -m shell -a "sudo systemctl stop filebeat && sudo apt-get install -y filebeat=8.12.0 && sudo systemctl start filebeat"

# Final verification
curl -s "localhost:9200/_cat/nodes?v&h=name,version,role"
echo "All nodes should show version 8.12.0"

curl -s "localhost:9200/_cluster/health" | jq '{status, number_of_nodes}'
echo "Status should be green with all 9 nodes"

What’s Next?

You’ve covered the essential production topics for the ELK Stack. Return to the Production Overview to review the chapter, or continue to the next chapter in the tutorial.

You may also enjoy