ELK Stack — Performance and Scaling Exercises

9 minute read

Practice Exercises

These exercises cover every topic in the Performance and Scaling chapter. Try each one on your own first, then expand the solution to check your work.

Exercise 1 — Design a Production Cluster Architecture

Scenario: You are building an Elasticsearch cluster for a company that ingests 200 GB of log data per day, retains data for 90 days, and runs dashboards in Kibana that query the last 7 days heavily. Older data is queried rarely.

Requirements:

  • High availability across 3 availability zones
  • Hot-warm-cold architecture
  • Dedicated masters
  • Coordinating nodes for Kibana

Tasks:

  1. Calculate total storage needed (with 1 replica)
  2. Determine the number and type of each node role
  3. Draw the cluster topology
  4. Write elasticsearch.yml snippets for one node of each role
Show Solution

Storage Calculation

Daily ingestion:       200 GB
Retention:             90 days
Total raw data:        200 × 90 = 18,000 GB = 18 TB
With 1 replica:        18 × 2 = 36 TB
Overhead (~15%):       36 × 1.15 ≈ 41.4 TB

Tier Breakdown

Tier Data Age Volume Storage Type Nodes
Hot 0–7 days ~2.8 TB (with replica) NVMe SSD 6 (2 per zone)
Warm 7–30 days ~9.2 TB (with replica) SSD/HDD 6 (2 per zone)
Cold 30–90 days ~24 TB (with replica) HDD/Object 4–6

Node Count

Role Count Sizing
Dedicated master 3 (1 per zone) 4 CPU, 8 GB RAM, 10 GB SSD
Hot data 6 (2 per zone) 16 CPU, 64 GB RAM, 2 TB NVMe
Warm data 6 (2 per zone) 8 CPU, 32 GB RAM, 4 TB SSD
Cold data 4 4 CPU, 16 GB RAM, 8 TB HDD
Coordinating 3 (1 per zone) 8 CPU, 16 GB RAM

Total: 22 nodes

Configuration Snippets

# Master node (master-01, zone a)
cluster.name: prod-logs
node.name: master-01
node.roles: [ master ]
node.attr.zone: us-east-1a
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: us-east-1a,us-east-1b,us-east-1c

# Hot data node (hot-01, zone a)
cluster.name: prod-logs
node.name: hot-01
node.roles: [ data_hot, ingest ]
node.attr.zone: us-east-1a
node.attr.box_type: hot

# Warm data node (warm-01, zone b)
cluster.name: prod-logs
node.name: warm-01
node.roles: [ data_warm ]
node.attr.zone: us-east-1b
node.attr.box_type: warm

# Cold data node (cold-01)
cluster.name: prod-logs
node.name: cold-01
node.roles: [ data_cold ]
node.attr.box_type: cold

# Coordinating node (coord-01, zone a)
cluster.name: prod-logs
node.name: coord-01
node.roles: [ ]
node.attr.zone: us-east-1a

Exercise 2 — Shard Strategy Calculation

Scenario: You have the following indices:

Index Daily Size Retention Current Primary Shards
logs-app 80 GB 30 days 5
logs-access 20 GB 30 days 5
metrics 5 GB 90 days 5
events 500 MB 365 days 5

Tasks:

  1. Calculate the optimal number of primary shards for each index
  2. Calculate total shard count across all indices
  3. Determine minimum heap needed to support these shards
  4. Identify which indices are over-sharded or under-sharded
Show Solution

Shard Calculation (target: 30 GB per shard)

Index Daily Size Optimal Shards Current Problem
logs-app 80 GB ceil(80/30) = 3 5 Slightly over-sharded
logs-access 20 GB 1 5 Over-sharded (4 GB per shard)
metrics 5 GB 1 5 Heavily over-sharded (1 GB per shard)
events 500 MB 1 5 Massively over-sharded (100 MB per shard)

Total Shard Count

With current shards (primary + 1 replica):
  logs-app:    5 × 2 × 30 days  = 300 shards
  logs-access: 5 × 2 × 30 days  = 300 shards
  metrics:     5 × 2 × 90 days  = 900 shards
  events:      5 × 2 × 365 days = 3,650 shards
  TOTAL:                           5,150 shards  ← TOO MANY

With optimized shards:
  logs-app:    3 × 2 × 30 days  = 180 shards
  logs-access: 1 × 2 × 30 days  = 60 shards
  metrics:     1 × 2 × 90 days  = 180 shards
  events:      1 × 2 × 365 days = 730 shards
  TOTAL:                           1,150 shards  ← Much better

Heap Calculation

At ~20 shards per GB of heap:
  Current:   5,150 / 20 = 257.5 GB heap needed  ← Impossible on one node
  Optimized: 1,150 / 20 = 57.5 GB heap needed   ← 2 nodes at 31 GB each

Recommendations

  1. Use data streams with rollover based on size (max_primary_shard_size: 30gb) instead of fixed shard counts
  2. For events (small daily volume), use a monthly or weekly rollover instead of daily
  3. Shrink old metrics and events indices to 1 shard each
  4. Set cluster.max_shards_per_node: 1000 as a safety net

Exercise 3 — JVM and System Tuning

Scenario: Your Elasticsearch node has:

  • 64 GB RAM
  • 16 CPU cores
  • NVMe SSD storage
  • Running RHEL 8

Operators report occasional long GC pauses (> 2 seconds) and circuit breaker exceptions during peak hours.

Tasks:

  1. Write the complete jvm.options.d/custom.options file
  2. Write the OS-level tuning commands
  3. Configure circuit breakers to prevent OOM
  4. Write the commands to verify all settings are applied
Show Solution

JVM Options

# /etc/elasticsearch/jvm.options.d/custom.options

## Heap — 50% of 64 GB, capped at 31 GB
-Xms31g
-Xmx31g

## G1GC tuning
-XX:+UseG1GC
-XX:G1HeapRegionSize=16m
-XX:InitiatingHeapOccupancyPercent=40
-XX:G1ReservePercent=15
-XX:MaxGCPauseMillis=200

## GC logging
-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

## Performance
-XX:+AlwaysPreTouch
-server

## Error handling
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

OS Tuning

# Disable swap
sudo swapoff -a
# Remove swap lines from /etc/fstab

# Virtual memory
sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf

# Swappiness
sudo sysctl -w vm.swappiness=1
echo "vm.swappiness=1" | sudo tee -a /etc/sysctl.conf

# File descriptors and threads
cat <<EOF | sudo tee /etc/security/limits.d/elasticsearch.conf
elasticsearch  -  nofile  65535
elasticsearch  -  nproc   4096
elasticsearch  -  memlock unlimited
EOF

# Memory lock in elasticsearch.yml
# bootstrap.memory_lock: true

# Reload sysctl
sudo sysctl -p

Circuit Breakers

curl -X PUT "localhost:9200/_cluster/settings" \
  -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "indices.breaker.fielddata.limit": "35%",
    "indices.breaker.request.limit": "55%",
    "indices.breaker.total.limit": "65%"
  }
}'

Verification Commands

# Heap settings
curl -s "localhost:9200/_nodes/stats/jvm?filter_path=nodes.*.jvm.mem.heap_max_in_bytes"

# Memory lock
curl -s "localhost:9200/_nodes?filter_path=**.mlockall"

# File descriptors
curl -s "localhost:9200/_nodes/stats/process?filter_path=**.max_file_descriptors"

# Circuit breaker settings
curl -s "localhost:9200/_nodes/stats/breaker?pretty"

# GC stats
curl -s "localhost:9200/_nodes/stats/jvm?filter_path=nodes.*.jvm.gc"

# OS memory
curl -s "localhost:9200/_nodes/stats/os?filter_path=nodes.*.os.mem"

Exercise 4 — Bulk Indexing Optimization

Scenario: You need to bulk-load 50 million documents (average 2 KB each, ~100 GB total) into a new index as fast as possible. Your cluster has 6 hot data nodes, each with 16 cores and 64 GB RAM.

Tasks:

  1. Create the index with optimal settings for bulk loading
  2. Write a bulk indexing strategy (payload size, parallelism, ID generation)
  3. Write the commands to restore normal settings after loading
  4. Run a force merge
Show Solution

Step 1: Create Index with Bulk-Optimized Settings

curl -X PUT "localhost:9200/bulk-load-index" \
  -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 6,
    "number_of_replicas": 0,
    "refresh_interval": "-1",
    "index.translog.durability": "async",
    "index.translog.sync_interval": "30s",
    "index.translog.flush_threshold_size": "1gb",
    "index.codec": "best_compression",
    "index.merge.scheduler.max_thread_count": 4
  },
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "message": { "type": "text" },
      "level": { "type": "keyword" },
      "service": { "type": "keyword" }
    }
  }
}'

Step 2: Bulk Indexing Strategy

Document size:  2 KB average
Payload target: 10 MB per bulk request
Docs per bulk:  10 MB / 2 KB ≈ 5,000 documents per request
Workers:        6 data nodes × 2 = 12 parallel bulk threads

# Sample bulk request — omit _id to let ES auto-generate (faster)
curl -X POST "localhost:9200/bulk-load-index/_bulk" \
  -H 'Content-Type: application/x-ndjson' -d'
{"index": {}}
{"@timestamp": "2026-04-01T10:00:00Z", "level": "INFO", "service": "api", "message": "Request processed"}
{"index": {}}
{"@timestamp": "2026-04-01T10:00:01Z", "level": "ERROR", "service": "api", "message": "Database timeout"}
'

Key points:

  • Omit _id — auto-generated IDs skip the version check (10–15% faster)
  • Use 12 parallel threads from the client
  • Monitor with curl -s localhost:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected
  • Back off if rejected count increases

Step 3: Restore Normal Settings

# Re-enable refresh
curl -X PUT "localhost:9200/bulk-load-index/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.refresh_interval": "1s",
  "index.number_of_replicas": 1,
  "index.translog.durability": "request"
}'

# Force a refresh to make all documents searchable
curl -X POST "localhost:9200/bulk-load-index/_refresh"

Step 4: Force Merge

# Wait for replica allocation to complete
curl -s "localhost:9200/_cat/recovery/bulk-load-index?v&active_only=true"

# Force merge to 1 segment per shard
curl -X POST "localhost:9200/bulk-load-index/_forcemerge?max_num_segments=1"

# Verify segment count
curl -s "localhost:9200/_cat/segments/bulk-load-index?v&h=index,shard,segment,size"

Expected Performance

50M docs × 2 KB = 100 GB
With 6 data nodes, 12 threads, 10 MB bulk:
  Throughput: ~50,000–100,000 docs/sec
  Time:       ~8–15 minutes

Exercise 5 — Query Profiling and Optimization

Scenario: Users report that the following Kibana dashboard query takes over 10 seconds:

{
  "size": 50,
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "payment failed" } },
        { "range": { "@timestamp": { "gte": "now-7d" } } },
        { "term": { "environment": "production" } },
        { "term": { "service.keyword": "checkout-api" } }
      ]
    }
  },
  "sort": [{ "@timestamp": "desc" }],
  "aggs": {
    "errors_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1h"
      }
    },
    "top_error_codes": {
      "terms": { "field": "error_code.keyword", "size": 10 }
    }
  }
}

Tasks:

  1. Identify all optimization problems in the query
  2. Rewrite the query with optimizations applied
  3. Set up slow log to catch similar issues
  4. Write the profile API call and explain what to look for
Show Solution

Problems Identified

# Problem Impact
1 range and term filters are in must (query context) Not cached, scored unnecessarily
2 Full _source returned Wasted network I/O for 50 docs
3 No filter_path or _source filter Returns aggregation buckets + all source fields
4 Aggregation spans entire result set Could be expensive on large matching sets
5 No timeout or terminate_after Query can run indefinitely

Optimized Query

curl -X GET "localhost:9200/logs-*/_search?timeout=5s" \
  -H 'Content-Type: application/json' -d'
{
  "size": 50,
  "_source": ["@timestamp", "message", "error_code", "service"],
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "payment failed" } }
      ],
      "filter": [
        { "range": { "@timestamp": { "gte": "now-7d" } } },
        { "term": { "environment": "production" } },
        { "term": { "service.keyword": "checkout-api" } }
      ]
    }
  },
  "sort": [{ "@timestamp": "desc" }],
  "aggs": {
    "errors_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1h"
      }
    },
    "top_error_codes": {
      "terms": { "field": "error_code.keyword", "size": 10 }
    }
  }
}'

Changes made:

  1. Moved range, term filters to filter context → cached, not scored
  2. Added _source filtering → only return needed fields
  3. Added timeout=5s → prevent runaway queries
  4. If index is sorted by @timestamp desc, the sort is free

Slow Log Configuration

curl -X PUT "localhost:9200/logs-*/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "2s",
  "index.search.slowlog.threshold.query.debug": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.search.slowlog.threshold.fetch.info": "500ms",
  "index.search.slowlog.level": "info"
}'

Profile API Call

curl -X GET "localhost:9200/logs-*/_search" \
  -H 'Content-Type: application/json' -d'
{
  "profile": true,
  "size": 0,
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "payment failed" } }
      ],
      "filter": [
        { "range": { "@timestamp": { "gte": "now-7d" } } },
        { "term": { "environment": "production" } },
        { "term": { "service.keyword": "checkout-api" } }
      ]
    }
  }
}'

What to look for in the profile output:

  1. time_in_nanos on each child — find the slowest clause
  2. build_scorer — high values mean the query structure is complex
  3. score time — if high, check if clauses belong in filter instead
  4. Number of shards queried — use routing or narrower index patterns
  5. Aggregation collectors — look for time_in_nanos on each agg to find expensive ones
  6. next_doc and advance — high values indicate the query matches too many documents; add more selective filters

Summary

Exercise Key Skills Practiced
1 Cluster topology design, node roles, zone awareness
2 Shard sizing, over/under-sharding detection, heap calculation
3 JVM options, OS tuning, circuit breakers, verification
4 Bulk API, refresh/replica tuning, force merge
5 Query profiling, filter context, slow log, optimization

← Query Optimization · Back to Performance and Scaling

You may also enjoy