ELK Stack — Elasticsearch Advanced Exercises

9 minute read

Elasticsearch Advanced — Practice Exercises

Put your knowledge of index templates, ILM, aliases, analyzers, and ingest pipelines to work. Each exercise builds on the concepts from this chapter. Try to solve each one before looking at the solution.

Exercise 1: Composable Index Template for Application Logs

Difficulty: ⭐⭐ Intermediate

Scenario

Your team generates application logs that are stored in indices named app-logs-YYYY.MM.DD. You need a standardized template so that every new daily index automatically gets the correct settings and mappings.

Requirements

Create a component template called app-log-settings with:
- 1 primary shard, 1 replica
- Refresh interval of 10s
Create a component template called app-log-mappings with these fields:
- @timestamp — date
- message — text
- log.level — keyword
- service.name — keyword
- trace.id — keyword
Create a composable index template called app-logs-template that:
- Matches app-logs-*
- Uses both component templates
- Has priority 200
- Adds an alias called app-logs
Verify your template by simulating index creation for app-logs-2026.04.01.

Show Solution

Step 1: Create the settings component template

curl -s -X PUT "localhost:9200/_component_template/app-log-settings" \
  -H "Content-Type: application/json" \
  -d '{
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "refresh_interval": "10s"
    }
  }
}'

{
  "acknowledged": true
}

Step 2: Create the mappings component template

curl -s -X PUT "localhost:9200/_component_template/app-log-mappings" \
  -H "Content-Type: application/json" \
  -d '{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp":   { "type": "date" },
        "message":      { "type": "text" },
        "log.level":    { "type": "keyword" },
        "service.name": { "type": "keyword" },
        "trace.id":     { "type": "keyword" }
      }
    }
  }
}'

{
  "acknowledged": true
}

Step 3: Create the composable index template

curl -s -X PUT "localhost:9200/_index_template/app-logs-template" \
  -H "Content-Type: application/json" \
  -d '{
  "index_patterns": ["app-logs-*"],
  "composed_of": ["app-log-settings", "app-log-mappings"],
  "priority": 200,
  "template": {
    "aliases": {
      "app-logs": {}
    }
  }
}'

{
  "acknowledged": true
}

Step 4: Simulate

curl -s -X POST "localhost:9200/_index_template/_simulate_index/app-logs-2026.04.01?pretty"

{
  "template": {
    "settings": {
      "index": {
        "number_of_shards": "1",
        "number_of_replicas": "1",
        "refresh_interval": "10s"
      }
    },
    "mappings": {
      "properties": {
        "@timestamp":   { "type": "date" },
        "message":      { "type": "text" },
        "log.level":    { "type": "keyword" },
        "service.name": { "type": "keyword" },
        "trace.id":     { "type": "keyword" }
      }
    },
    "aliases": {
      "app-logs": {}
    }
  }
}

Exercise 2: ILM Policy for Metrics Data

Difficulty: ⭐⭐⭐ Advanced

Scenario

Your monitoring system stores metrics in Elasticsearch. You need an ILM policy that automatically manages the index lifecycle to control costs.

Requirements

Create an ILM policy called metrics-policy with these phases:
- Hot: Rollover when the index reaches 25 GB or is 1 day old. Set priority to 100.
- Warm: After 3 days, shrink to 1 shard, force-merge to 1 segment, set priority to 50.
- Cold: After 30 days, allocate to nodes with attribute data: cold.
- Delete: After 90 days, delete the index.
Create an index template called metrics-template that:
- Matches metrics-*
- References metrics-policy
- Sets metrics-write as the rollover alias
Bootstrap the first index metrics-000001 with the write alias.
Verify ILM is attached using _ilm/explain.

Show Solution

Step 1: Create the ILM policy

curl -s -X PUT "localhost:9200/_ilm/policy/metrics-policy" \
  -H "Content-Type: application/json" \
  -d '{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "25gb",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": { "data": "cold" }
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}'

{
  "acknowledged": true
}

Step 2: Create the index template

curl -s -X PUT "localhost:9200/_index_template/metrics-template" \
  -H "Content-Type: application/json" \
  -d '{
  "index_patterns": ["metrics-*"],
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "index.lifecycle.name": "metrics-policy",
      "index.lifecycle.rollover_alias": "metrics-write"
    }
  },
  "priority": 200
}'

{
  "acknowledged": true
}

Step 3: Bootstrap the first index

curl -s -X PUT "localhost:9200/metrics-000001" \
  -H "Content-Type: application/json" \
  -d '{
  "aliases": {
    "metrics-write": {
      "is_write_index": true
    }
  }
}'

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "metrics-000001"
}

Step 4: Verify ILM

curl -s -X GET "localhost:9200/metrics-000001/_ilm/explain?pretty"

{
  "indices": {
    "metrics-000001": {
      "index": "metrics-000001",
      "managed": true,
      "policy": "metrics-policy",
      "phase": "hot",
      "action": "complete",
      "step": "complete"
    }
  }
}

Exercise 3: Zero-Downtime Reindex with Aliases

Difficulty: ⭐⭐ Intermediate

Scenario

You have a customers index (v1) with a text field for email. You realize email should be keyword for exact matching. Perform a zero-downtime migration using aliases.

Requirements

Create customers-v1 with name (text), email (text), and country (keyword).
Point alias customers to customers-v1 as the write index.
Index two sample documents through the alias.
Create customers-v2 with email changed to keyword.
Reindex from v1 to v2.
Atomically swap the alias from v1 to v2.
Verify the alias points to v2.

Show Solution

Step 1: Create v1

curl -s -X PUT "localhost:9200/customers-v1" \
  -H "Content-Type: application/json" \
  -d '{
  "mappings": {
    "properties": {
      "name":    { "type": "text" },
      "email":   { "type": "text" },
      "country": { "type": "keyword" }
    }
  }
}'

Step 2: Create alias

curl -s -X POST "localhost:9200/_aliases" \
  -H "Content-Type: application/json" \
  -d '{
  "actions": [
    { "add": { "index": "customers-v1", "alias": "customers", "is_write_index": true } }
  ]
}'

Step 3: Index sample documents

curl -s -X POST "localhost:9200/customers/_doc/1" \
  -H "Content-Type: application/json" \
  -d '{ "name": "Alice Smith", "email": "alice@example.com", "country": "US" }'

curl -s -X POST "localhost:9200/customers/_doc/2" \
  -H "Content-Type: application/json" \
  -d '{ "name": "Bob Jones", "email": "bob@example.com", "country": "UK" }'

Step 4: Create v2 with corrected mapping

curl -s -X PUT "localhost:9200/customers-v2" \
  -H "Content-Type: application/json" \
  -d '{
  "mappings": {
    "properties": {
      "name":    { "type": "text" },
      "email":   { "type": "keyword" },
      "country": { "type": "keyword" }
    }
  }
}'

Step 5: Reindex

curl -s -X POST "localhost:9200/_reindex" \
  -H "Content-Type: application/json" \
  -d '{
  "source": { "index": "customers-v1" },
  "dest":   { "index": "customers-v2" }
}'

{
  "took": 42,
  "total": 2,
  "created": 2,
  "updated": 0,
  "deleted": 0,
  "failures": []
}

Step 6: Atomic alias swap

curl -s -X POST "localhost:9200/_aliases" \
  -H "Content-Type: application/json" \
  -d '{
  "actions": [
    { "remove": { "index": "customers-v1", "alias": "customers" } },
    { "add":    { "index": "customers-v2", "alias": "customers", "is_write_index": true } }
  ]
}'

{
  "acknowledged": true
}

Step 7: Verify

curl -s -X GET "localhost:9200/_alias/customers?pretty"

{
  "customers-v2": {
    "aliases": {
      "customers": {
        "is_write_index": true
      }
    }
  }
}

Exercise 4: Custom Analyzer for Product Search

Difficulty: ⭐⭐⭐ Advanced

Scenario

You are building a product search for an e-commerce site. Products have names in multiple languages and may contain HTML from the CMS. You need a custom analyzer that handles this.

Requirements

Create an index called product-catalog with a custom analyzer named product_analyzer that:
- Strips HTML tags (character filter)
- Uses the standard tokenizer
- Lowercases all tokens
- Removes English stop words
- Applies ASCII folding (so café matches cafe)
- Applies English stemming
Map the name field to use product_analyzer.
Map description to use product_analyzer.
Test the analyzer using the _analyze API with the input: "<b>Premium Café</b> Machines — Pro Series"

Show Solution

Step 1: Create the index with custom analyzer

curl -s -X PUT "localhost:9200/product-catalog" \
  -H "Content-Type: application/json" \
  -d '{
  "settings": {
    "analysis": {
      "char_filter": {
        "strip_html": {
          "type": "html_strip"
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        },
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        }
      },
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "char_filter": ["strip_html"],
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "english_stop",
            "asciifolding",
            "english_stemmer"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name":        { "type": "text", "analyzer": "product_analyzer" },
      "description": { "type": "text", "analyzer": "product_analyzer" },
      "sku":         { "type": "keyword" },
      "price":       { "type": "scaled_float", "scaling_factor": 100 }
    }
  }
}'

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "product-catalog"
}

Step 2: Test the analyzer

curl -s -X POST "localhost:9200/product-catalog/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "product_analyzer",
  "text": "<b>Premium Café</b> Machines — Pro Series"
}'

{
  "tokens": [
    { "token": "premium",  "start_offset": 3,  "end_offset": 14, "type": "<ALPHANUM>", "position": 0 },
    { "token": "cafe",     "start_offset": 15, "end_offset": 23, "type": "<ALPHANUM>", "position": 1 },
    { "token": "machin",   "start_offset": 28, "end_offset": 36, "type": "<ALPHANUM>", "position": 2 },
    { "token": "pro",      "start_offset": 39, "end_offset": 42, "type": "<ALPHANUM>", "position": 3 },
    { "token": "seri",     "start_offset": 43, "end_offset": 49, "type": "<ALPHANUM>", "position": 4 }
  ]
}

HTML is stripped, Café → cafe (ASCII folded), Machines → machin (stemmed), Series → seri (stemmed). Stop words like any articles would be removed.

Exercise 5: Ingest Pipeline for Web Server Logs

Difficulty: ⭐⭐⭐⭐ Expert

Scenario

Your Nginx web servers send access logs to Elasticsearch. You need an ingest pipeline that parses these logs, extracts fields, converts types, adds geo-enrichment, and handles parsing failures gracefully.

Requirements

Create a pipeline called nginx-access-pipeline with these processors:
- grok: Parse the combined log format from the message field
- date: Parse the timestamp into @timestamp
- convert: Convert http.response.status_code to integer and http.response.body.bytes to long
- user_agent: Parse the user_agent.original field
- geoip: Enrich source.ip into source.geo — but only for non-private IPs
- set: Add event.kind = "event" and event.category = "web"
- remove: Remove the original message and event.raw_timestamp fields
Add an on_failure handler that:
- Sets event.kind to "pipeline_error"
- Captures the error message

Simulate the pipeline with this sample log line:

203.0.113.50 - admin [01/Apr/2026:10:15:30 +0530] "POST /api/orders HTTP/1.1" 201 512 "https://shop.example.com/cart" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"

Verify the simulation output has all expected fields.

Show Solution

Step 1: Create the pipeline

curl -s -X PUT "localhost:9200/_ingest/pipeline/nginx-access-pipeline" \
  -H "Content-Type: application/json" \
  -d '{
  "description": "Parse Nginx combined access logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{IPORHOST:source.ip} - %{DATA:user.name} \\[%{HTTPDATE:event.raw_timestamp}\\] \"%{WORD:http.method} %{DATA:url.path} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.response.status_code} %{NUMBER:http.response.body.bytes} \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\""
        ]
      }
    },
    {
      "date": {
        "field": "event.raw_timestamp",
        "formats": ["dd/MMM/yyyy:HH:mm:ss Z"],
        "target_field": "@timestamp"
      }
    },
    {
      "convert": {
        "field": "http.response.status_code",
        "type": "integer"
      }
    },
    {
      "convert": {
        "field": "http.response.body.bytes",
        "type": "long"
      }
    },
    {
      "user_agent": {
        "field": "user_agent.original"
      }
    },
    {
      "geoip": {
        "field": "source.ip",
        "target_field": "source.geo",
        "if": "ctx.source?.ip != null && !ctx.source.ip.startsWith(\"192.168.\") && !ctx.source.ip.startsWith(\"10.\")"
      }
    },
    {
      "set": {
        "field": "event.kind",
        "value": "event"
      }
    },
    {
      "set": {
        "field": "event.category",
        "value": "web"
      }
    },
    {
      "remove": {
        "field": ["message", "event.raw_timestamp"]
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "event.kind",
        "value": "pipeline_error"
      }
    },
    {
      "set": {
        "field": "error.message",
        "value": "{{_ingest.on_failure_message}}"
      }
    }
  ]
}'

{
  "acknowledged": true
}

Step 2: Simulate with a sample log

curl -s -X POST "localhost:9200/_ingest/pipeline/nginx-access-pipeline/_simulate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "docs": [
    {
      "_source": {
        "message": "203.0.113.50 - admin [01/Apr/2026:10:15:30 +0530] \"POST /api/orders HTTP/1.1\" 201 512 \"https://shop.example.com/cart\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)\""
      }
    }
  ]
}'

{
  "docs": [
    {
      "doc": {
        "_source": {
          "@timestamp": "2026-04-01T04:45:30.000Z",
          "source": {
            "ip": "203.0.113.50",
            "geo": {
              "continent_name": "North America",
              "country_iso_code": "US",
              "location": { "lat": 37.751, "lon": -97.822 }
            }
          },
          "user": { "name": "admin" },
          "http": {
            "method": "POST",
            "version": "1.1",
            "request": {
              "referrer": "https://shop.example.com/cart"
            },
            "response": {
              "status_code": 201,
              "body": { "bytes": 512 }
            }
          },
          "url": { "path": "/api/orders" },
          "user_agent": {
            "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)",
            "os": { "name": "Mac OS X", "version": "10.15.7" },
            "name": "Other"
          },
          "event": {
            "kind": "event",
            "category": "web"
          }
        },
        "_ingest": {
          "timestamp": "2026-04-01T09:00:00.000Z"
        }
      }
    }
  ]
}

The pipeline successfully:

Parsed the combined log format with grok
Converted the timestamp to @timestamp
Converted status code to integer and bytes to long
Parsed the user agent string
Added GeoIP data for the public IP
Set event metadata fields
Removed the raw message and event.raw_timestamp fields

What’s Next

Congratulations on completing the Elasticsearch Advanced chapter! You now know how to:

Standardize index creation with composable templates
Automate index lifecycle with ILM policies
Use aliases for zero-downtime operations
Build custom analyzers for precise text analysis
Transform data at ingest time with pipelines

Move on to the next chapter to continue building your ELK Stack skills.

Share on

Twitter Facebook LinkedIn

ELK Stack — Elasticsearch Advanced Exercises

Elasticsearch Advanced — Practice Exercises

Exercise 1: Composable Index Template for Application Logs

Scenario

Requirements

Exercise 2: ILM Policy for Metrics Data

Scenario

Requirements

Exercise 3: Zero-Downtime Reindex with Aliases

Scenario

Requirements

Exercise 4: Custom Analyzer for Product Search

Scenario

Requirements

Exercise 5: Ingest Pipeline for Web Server Logs

Scenario

Requirements

What’s Next

Share on

You may also enjoy

AWS EKS Karpenter: Intelligent Kubernetes Node Autoscaling and Provisioning

Terraform Provider Development: Building Custom Providers with the Plugin Framework

EKS IRSA and Pod Identity: Fine-Grained AWS Permissions Without Static Keys

Writing Resume: Quantifying Infrastructure Impact with Metrics and Cost Savings