ELK Stack — Query DSL Basics

6 minute read

Introduction

The Query DSL (Domain Specific Language) is the JSON-based language you use to define searches in Elasticsearch. Every search request — whether it comes from Kibana, Logstash, or a curl command — ultimately speaks Query DSL under the hood.

Understanding how the Query DSL is structured, what contexts exist, and how scoring works is the foundation for writing effective queries.

Query DSL Structure

Every search request follows a consistent anatomy. Here is the high-level structure:

┌─────────────────────────────────────────────────────────┐
│                   Search Request                        │
├─────────────────────────────────────────────────────────┤
│  GET /<index>/_search                                   │
│  {                                                      │
│    "query": { ... },          ← What to match           │
│    "sort": [ ... ],           ← How to order results    │
│    "from": 0, "size": 10,    ← Pagination               │
│    "aggs": { ... },           ← Aggregations            │
│    "_source": [ ... ],        ← Fields to return        │
│    "post_filter": { ... }     ← Filter after aggs       │
│  }                                                      │
├─────────────────────────────────────────────────────────┤
│              Search Response                            │
│  {                                                      │
│    "took": 5,                 ← Time in ms              │
│    "hits": {                                            │
│      "total": { "value": N },                           │
│      "max_score": 1.2,                                  │
│      "hits": [ ... ]          ← Matched documents       │
│    },                                                   │
│    "aggregations": { ... }    ← Agg results             │
│  }                                                      │
└─────────────────────────────────────────────────────────┘

Setting Up Sample Data

Before running the examples, let’s create a sample web-logs index:

curl -s -X PUT "localhost:9200/web-logs" -H 'Content-Type: application/json' -d '{
  "mappings": {
    "properties": {
      "method":        { "type": "keyword" },
      "url":           { "type": "text", "fields": { "raw": { "type": "keyword" } } },
      "status_code":   { "type": "integer" },
      "response_time": { "type": "float" },
      "client_ip":     { "type": "ip" },
      "message":       { "type": "text" },
      "user_agent":    { "type": "text", "fields": { "raw": { "type": "keyword" } } },
      "@timestamp":    { "type": "date" }
    }
  }
}'

Now bulk-index some sample documents:

curl -s -X POST "localhost:9200/web-logs/_bulk" -H 'Content-Type: application/json' -d '
{"index":{}}
{"method":"GET","url":"/api/users","status_code":200,"response_time":120.5,"client_ip":"192.168.1.10","message":"Successfully fetched user list","user_agent":"Mozilla/5.0","@timestamp":"2026-04-01T08:00:00Z"}
{"index":{}}
{"method":"POST","url":"/api/users","status_code":201,"response_time":350.2,"client_ip":"192.168.1.11","message":"New user created successfully","user_agent":"curl/7.68.0","@timestamp":"2026-04-01T08:05:00Z"}
{"index":{}}
{"method":"GET","url":"/api/products","status_code":200,"response_time":95.3,"client_ip":"192.168.1.10","message":"Product catalog returned","user_agent":"Mozilla/5.0","@timestamp":"2026-04-01T08:10:00Z"}
{"index":{}}
{"method":"GET","url":"/api/users/42","status_code":404,"response_time":15.1,"client_ip":"10.0.0.5","message":"User not found error occurred","user_agent":"PostmanRuntime/7.29","@timestamp":"2026-04-01T08:15:00Z"}
{"index":{}}
{"method":"DELETE","url":"/api/users/7","status_code":500,"response_time":5020.0,"client_ip":"10.0.0.5","message":"Internal server error during delete operation","user_agent":"PostmanRuntime/7.29","@timestamp":"2026-04-01T08:20:00Z"}
{"index":{}}
{"method":"GET","url":"/api/orders","status_code":200,"response_time":210.7,"client_ip":"172.16.0.1","message":"Order history fetched for user","user_agent":"Mozilla/5.0","@timestamp":"2026-04-01T09:00:00Z"}
'

The _search Endpoint

The _search endpoint is your primary interface for querying Elasticsearch. It accepts both GET and POST methods.

Basic Search — Return All Documents

curl -s -X GET "localhost:9200/web-logs/_search?pretty"

Response:

{
  "took": 3,
  "timed_out": false,
  "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 },
  "hits": {
    "total": { "value": 6, "relation": "eq" },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "web-logs",
        "_id": "abc123",
        "_score": 1.0,
        "_source": {
          "method": "GET",
          "url": "/api/users",
          "status_code": 200,
          "response_time": 120.5,
          "message": "Successfully fetched user list",
          "@timestamp": "2026-04-01T08:00:00Z"
        }
      }
    ]
  }
}

Explicit match_all Query

The match_all query matches every document and assigns a score of 1.0:

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": {
    "match_all": {}
  }
}'

{
  "took": 2,
  "hits": {
    "total": { "value": 6, "relation": "eq" },
    "max_score": 1.0,
    "hits": [ "... all 6 documents ..." ]
  }
}

Query Context vs Filter Context

This is one of the most important concepts in Elasticsearch querying. The context in which a query clause is used determines whether it contributes to scoring.

Query Context

Clauses in query context answer: “How well does this document match?”

  • Elasticsearch calculates a relevance score (_score)
  • Results are ranked by score (highest first)
  • Used inside the query parameter or bool.must / bool.should
curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": {
    "match": {
      "message": "user error"
    }
  }
}'

{
  "took": 4,
  "hits": {
    "total": { "value": 4, "relation": "eq" },
    "max_score": 1.7917595,
    "hits": [
      {
        "_id": "...",
        "_score": 1.7917595,
        "_source": { "message": "User not found error occurred", "status_code": 404 }
      },
      {
        "_id": "...",
        "_score": 1.5404451,
        "_source": { "message": "Internal server error during delete operation", "status_code": 500 }
      },
      {
        "_id": "...",
        "_score": 0.6931471,
        "_source": { "message": "Successfully fetched user list", "status_code": 200 }
      },
      {
        "_id": "...",
        "_score": 0.5753642,
        "_source": { "message": "New user created successfully", "status_code": 201 }
      }
    ]
  }
}

Notice how documents containing both “user” and “error” score higher than those with only one term.

Filter Context

Clauses in filter context answer: “Does this document match? Yes or No.”

  • No scoring — documents either match or don’t
  • Faster because Elasticsearch can cache filter results
  • Used inside bool.filter or bool.must_not
curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": {
    "bool": {
      "filter": [
        { "term": { "method": "GET" } }
      ]
    }
  }
}'

{
  "took": 1,
  "hits": {
    "total": { "value": 4, "relation": "eq" },
    "max_score": 0.0,
    "hits": [
      {
        "_id": "...",
        "_score": 0.0,
        "_source": { "method": "GET", "url": "/api/users" }
      }
    ]
  }
}

Notice _score is 0.0 — the filter determined which documents match, but did not rank them.

Full-Text vs Term-Level Queries

Elasticsearch provides two broad categories of queries:

Full-Text Queries

Full-text queries analyze the search input the same way the field was analyzed at index time. The input text goes through the analyzer pipeline (tokenization, lowercasing, stemming, etc.).

Query Purpose
match Standard full-text search
match_phrase Exact phrase in order
multi_match Search across multiple fields

Term-Level Queries

Term-level queries look for exact values in the inverted index. The search input is not analyzed — it must match exactly what’s stored.

Query Purpose
term Exact single value
terms Match any of several values
range Numeric/date ranges
exists Field exists
wildcard Pattern matching

query vs post_filter

Both narrow down results, but they differ in when they apply:

  • query — Filters documents before aggregations run. Aggregations see only the matched documents.
  • post_filter — Filters documents after aggregations run. Aggregations see the full dataset, but the returned hits are filtered.

post_filter is useful for faceted navigation (e.g., filtering search results by color while still showing counts for all colors).

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": {
    "match_all": {}
  },
  "aggs": {
    "methods": {
      "terms": { "field": "method" }
    }
  },
  "post_filter": {
    "term": { "method": "GET" }
  }
}'

{
  "took": 5,
  "hits": {
    "total": { "value": 4, "relation": "eq" },
    "hits": [ "... only GET documents ..." ]
  },
  "aggregations": {
    "methods": {
      "buckets": [
        { "key": "GET", "doc_count": 4 },
        { "key": "POST", "doc_count": 1 },
        { "key": "DELETE", "doc_count": 1 }
      ]
    }
  }
}

The aggregation counts include all methods, but hits only returns GET requests.

Scoring and Relevance Basics

Elasticsearch uses the BM25 algorithm (Best Matching 25) to calculate relevance scores. Three factors drive the score:

  1. Term Frequency (TF) — How often does the term appear in the document? More occurrences = higher score (with diminishing returns).
  2. Inverse Document Frequency (IDF) — How rare is the term across all documents? Rarer terms are weighted higher.
  3. Field Length Norm — Shorter fields get higher scores. A match in a 3-word title scores higher than the same match in a 500-word body.

Explain API

Use explain=true to see how a score was calculated:

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "explain": true,
  "query": {
    "match": {
      "message": "error"
    }
  },
  "size": 1
}'

{
  "hits": {
    "hits": [
      {
        "_id": "...",
        "_score": 1.1631508,
        "_explanation": {
          "value": 1.1631508,
          "description": "weight(message:error in 3) [PerFieldSimilarity], result of:",
          "details": [
            {
              "value": 1.1631508,
              "description": "score(freq=1.0), computed as boost * idf * tf",
              "details": [
                { "value": 2.2, "description": "boost" },
                { "value": 1.1631508, "description": "idf, computed as ..." },
                { "value": 0.45454544, "description": "tf, computed as ..." }
              ]
            }
          ]
        }
      }
    ]
  }
}

Quick Reference

Concept Description
_search endpoint Primary search API — GET /index/_search
match_all Matches every document with score 1.0
Query context Calculates relevance score, ranks results
Filter context Yes/No matching, no scoring, cacheable
Full-text queries Input is analyzed (match, match_phrase, multi_match)
Term-level queries Input is NOT analyzed (term, range, exists)
query Filters before aggregations
post_filter Filters after aggregations (hits only)
BM25 Default scoring algorithm (TF × IDF × field length)
explain: true Shows score breakdown for debugging

Next: Match and Term Queries — Deep dive into full-text and exact-value query types.

You may also enjoy