ELK Stack — Query Optimization

5 minute read

Query Optimization

Indexing gets data in. Searching gets answers out. This section covers every technique for making your queries faster — from filter caching and _source filtering to the profile API and slow log analysis.

Filter vs. Query Context

Understanding the difference between filter and query context is the single most important optimization.

Context Scores? Cached? Use When
query Yes No Full-text search, relevance ranking
filter No Yes Exact matches, ranges, existence checks

Move Filters into filter Clauses

# BAD — range in query context (scored, not cached)
curl -X GET "localhost:9200/logs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "error" } },
        { "range": { "@timestamp": { "gte": "2026-04-01" } } }
      ]
    }
  }
}'

# GOOD — range in filter context (not scored, cached)
curl -X GET "localhost:9200/logs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "error" } }
      ],
      "filter": [
        { "range": { "@timestamp": { "gte": "2026-04-01" } } }
      ]
    }
  }
}'

Rule of thumb: If you do not need relevance scoring for a clause, put it in filter.

Bool Query Optimization

Structure your bool queries to maximize filter cache hits:

curl -X GET "localhost:9200/logs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "timeout" } }
      ],
      "filter": [
        { "term": { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "2026-03-01", "lte": "2026-04-01" } } },
        { "term": { "service": "payment-api" } }
      ],
      "must_not": [
        { "term": { "environment": "staging" } }
      ]
    }
  }
}'

Optimization tips:

  1. Put the most selective filter first — Elasticsearch evaluates filters in order and skips documents early
  2. Use term / terms instead of match for keyword fields
  3. Avoid wildcard with leading wildcards (*error) — it scans every term
  4. Prefer bool + filter over constant_score for multiple clauses

_source Filtering

By default, Elasticsearch returns the entire _source document. If you only need a few fields, filtering _source reduces network I/O and serialization time.

# Return only specific fields
curl -X GET "localhost:9200/logs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "size": 100,
  "_source": ["@timestamp", "level", "message"],
  "query": {
    "bool": {
      "filter": [
        { "term": { "level": "ERROR" } }
      ]
    }
  }
}'

# Exclude large fields
curl -X GET "localhost:9200/logs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "_source": {
    "excludes": ["stacktrace", "request_body"]
  },
  "query": { "match_all": {} }
}'

For aggregation-only queries, disable _source entirely:

curl -X GET "localhost:9200/logs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "_source": false,
  "aggs": {
    "errors_per_service": {
      "terms": { "field": "service.keyword", "size": 20 }
    }
  }
}'

Doc Values

Doc values are the columnar data structure used for sorting, aggregations, and scripting. They are enabled by default for all non-text fields.

  • Aggregation fields — must have doc_values: true (default)
  • Fields never aggregated or sorted — disable doc_values to save disk space
curl -X PUT "localhost:9200/logs" \
  -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "message": {
        "type": "text",
        "doc_values": false
      },
      "level": {
        "type": "keyword"
      },
      "request_id": {
        "type": "keyword",
        "doc_values": false,
        "index": true
      }
    }
  }
}'

Profile API

The profile API reveals exactly how long each component of your query takes.

curl -X GET "localhost:9200/logs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "profile": true,
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "connection timeout" } }
      ],
      "filter": [
        { "term": { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "2026-04-01" } } }
      ]
    }
  }
}'

Interpreting Profile Output

{
  "profile": {
    "shards": [
      {
        "searches": [
          {
            "query": [
              {
                "type": "BooleanQuery",
                "description": "+message:connection +message:timeout #level:ERROR #@timestamp:[2026-04-01 TO *]",
                "time_in_nanos": 1250000,
                "breakdown": {
                  "score": 450000,
                  "build_scorer": 320000,
                  "next_doc": 280000,
                  "advance": 50000,
                  "match": 0,
                  "create_weight": 150000
                },
                "children": [
                  {
                    "type": "TermQuery",
                    "description": "message:connection",
                    "time_in_nanos": 380000
                  },
                  {
                    "type": "TermQuery",
                    "description": "message:timeout",
                    "time_in_nanos": 350000
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}

Key fields:

Field Meaning
time_in_nanos Total time for this query component
score Time spent calculating relevance scores
build_scorer Time building the scorer iterator
next_doc Time advancing to the next matching doc
create_weight Time creating the query weight (one-time cost)

Explain API

Use _explain to understand why a specific document matched (or did not match) a query:

curl -X GET "localhost:9200/logs/_explain/abc123" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": { "message": "timeout" }
  }
}'

Slow Log

The search slow log records queries that exceed a time threshold. This is your primary tool for finding problematic queries in production.

curl -X PUT "localhost:9200/logs/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "2s",
  "index.search.slowlog.threshold.query.debug": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.search.slowlog.threshold.fetch.info": "500ms",
  "index.search.slowlog.level": "info"
}'

Slow log entries appear in the Elasticsearch log file:

[WARN ][index.search.slowlog] [hot-01] [logs-2026.04][0] took[5.2s], took_millis[5200], total_hits[45000], source[{"query":{"match_all":{}}}]

Indexing Slow Log

curl -X PUT "localhost:9200/logs/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.indexing.slowlog.threshold.index.warn": "10s",
  "index.indexing.slowlog.threshold.index.info": "5s"
}'

Search Timeout and Terminate After

Protect your cluster from runaway queries:

# Timeout — return partial results after 5 seconds
curl -X GET "localhost:9200/logs/_search?timeout=5s" \
  -H 'Content-Type: application/json' -d'
{
  "query": { "match": { "message": "error" } }
}'

# terminate_after — stop after collecting N docs per shard
curl -X GET "localhost:9200/logs/_search" \
  -H 'Content-Type: application/json' -d'
{
  "terminate_after": 1000,
  "query": { "match": { "message": "error" } }
}'

The response includes "timed_out": true or "terminated_early": true so the client knows results are partial.

Search Routing

If your data is routed by a specific key, include that key in your search to hit only one shard instead of all shards:

# Search only the shard holding customer_42's data
curl -X GET "localhost:9200/orders/_search?routing=customer_42" \
  -H 'Content-Type: application/json' -d'
{
  "query": { "term": { "customer_id": "customer_42" } }
}'

Index Sorting

Pre-sorting data on disk can dramatically speed up queries that sort by the same field:

curl -X PUT "localhost:9200/events" \
  -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.sort.field": "@timestamp",
    "index.sort.order": "desc"
  },
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "level": { "type": "keyword" }
    }
  }
}'

Note: Index sorting slows down indexing because documents must be inserted in sorted order. Use it only when your primary query pattern always sorts on the same field.

Searchable Snapshots

Searchable snapshots let you query data stored in a snapshot repository (S3, GCS, Azure Blob) without fully restoring it. Data is fetched on demand and cached locally.

# Mount a searchable snapshot
curl -X POST "localhost:9200/_snapshot/my-repo/snapshot-2026.03/_mount?storage=shared_cache" \
  -H 'Content-Type: application/json' -d'
{
  "index": "logs-2026.03",
  "renamed_index": "restored-logs-2026.03"
}'

Use searchable snapshots for the frozen tier in hot-warm-cold architecture to dramatically reduce storage costs.

Quick Reference

Technique When to Use
Filter context Exact matches, ranges, existence — no scoring needed
_source filtering Reduce payload size for large documents
Disable doc_values Fields never sorted or aggregated
Profile API Debug slow queries in development
Slow log Find slow queries in production
timeout Prevent runaway queries
terminate_after Cap results per shard for sampling
Search routing Queries always filter by a routing key
Index sorting Queries always sort by the same field
Searchable snapshots Query old data without full restore

← Indexing Performance · Next → Practice Exercises

You may also enjoy