ELK Stack — Sorting and Pagination

8 minute read

Introduction

By default, Elasticsearch returns search results sorted by relevance score (_score). But in many real-world scenarios — log analysis, time-series data, admin dashboards — you need custom sort orders and efficient pagination through large result sets.

This page covers sorting options, three pagination strategies (and when to use each), and techniques for controlling which fields are returned.

Sorting

Basic Sort

Sort by a single field in ascending or descending order:

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "sort": [
    { "response_time": "desc" }
  ]
}'

{
  "took": 2,
  "hits": {
    "total": { "value": 6, "relation": "eq" },
    "max_score": null,
    "hits": [
      { "sort": [5020.0], "_source": { "response_time": 5020.0, "method": "DELETE", "status_code": 500 } },
      { "sort": [350.2], "_source": { "response_time": 350.2, "method": "POST", "status_code": 201 } },
      { "sort": [210.7], "_source": { "response_time": 210.7, "method": "GET", "status_code": 200 } },
      { "sort": [120.5], "_source": { "response_time": 120.5, "method": "GET", "status_code": 200 } },
      { "sort": [95.3], "_source": { "response_time": 95.3, "method": "GET", "status_code": 200 } },
      { "sort": [15.1], "_source": { "response_time": 15.1, "method": "GET", "status_code": 404 } }
    ]
  }
}

Notice: When sorting by a field other than _score, max_score is null and each hit includes a sort array.

Multi-Field Sorting

Sort by multiple fields — Elasticsearch applies them in order:

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "sort": [
    { "status_code": "desc" },
    { "response_time": "asc" }
  ]
}'

{
  "hits": {
    "hits": [
      { "sort": [500, 5020.0], "_source": { "status_code": 500, "response_time": 5020.0 } },
      { "sort": [404, 15.1], "_source": { "status_code": 404, "response_time": 15.1 } },
      { "sort": [201, 350.2], "_source": { "status_code": 201, "response_time": 350.2 } },
      { "sort": [200, 95.3], "_source": { "status_code": 200, "response_time": 95.3 } },
      { "sort": [200, 120.5], "_source": { "status_code": 200, "response_time": 120.5 } },
      { "sort": [200, 210.7], "_source": { "status_code": 200, "response_time": 210.7 } }
    ]
  }
}

Documents with the same status_code are further sorted by response_time ascending.

Sort by _score

You can include _score as a sort field to combine relevance with other criteria:

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": {
    "match": { "message": "user" }
  },
  "sort": [
    { "status_code": "desc" },
    { "_score": "desc" }
  ]
}'

{
  "hits": {
    "hits": [
      { "sort": [404, 0.6931471], "_source": { "status_code": 404, "message": "User not found error occurred" } },
      { "sort": [201, 0.5753642], "_source": { "status_code": 201, "message": "New user created successfully" } },
      { "sort": [200, 0.6931471], "_source": { "status_code": 200, "message": "Successfully fetched user list" } },
      { "sort": [200, 0.5753642], "_source": { "status_code": 200, "message": "Order history fetched for user" } }
    ]
  }
}

Sort with Missing Values

Control where documents with missing field values appear:

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "sort": [
    {
      "response_time": {
        "order": "asc",
        "missing": "_last"
      }
    }
  ]
}'

Options for missing: _last, _first, or a specific value.

Pagination Strategies

Elasticsearch offers three pagination approaches, each suited to different use cases.

Strategy 1: from / size (Offset Pagination)

The simplest approach — skip from documents and return size documents:

# Page 1: documents 1-5
curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "sort": [{ "@timestamp": "desc" }],
  "from": 0,
  "size": 5
}'

{
  "hits": {
    "total": { "value": 6, "relation": "eq" },
    "hits": [
      { "sort": [1743498000000], "_source": { "@timestamp": "2026-04-01T09:00:00Z" } },
      { "sort": [1743495600000], "_source": { "@timestamp": "2026-04-01T08:20:00Z" } },
      { "sort": [1743495300000], "_source": { "@timestamp": "2026-04-01T08:15:00Z" } },
      { "sort": [1743495000000], "_source": { "@timestamp": "2026-04-01T08:10:00Z" } },
      { "sort": [1743494700000], "_source": { "@timestamp": "2026-04-01T08:05:00Z" } }
    ]
  }
}

# Page 2: documents 6-10
curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "sort": [{ "@timestamp": "desc" }],
  "from": 5,
  "size": 5
}'

{
  "hits": {
    "total": { "value": 6, "relation": "eq" },
    "hits": [
      { "sort": [1743494400000], "_source": { "@timestamp": "2026-04-01T08:00:00Z" } }
    ]
  }
}

Common Mistake: Deep Pagination with from / size

Warning: Elasticsearch has a hard limit of from + size <= 10,000 (configurable via index.max_result_window). Beyond this, you’ll get an error.

Why? With from: 9990, size: 10, each shard must return its top 10,000 results to the coordinating node, which then globally sorts and picks the final 10. This gets exponentially expensive.

# This will FAIL if from + size > 10000
curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "from": 10000,
  "size": 10
}'

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Result window is too large, from + size must be less than or equal to: [10000]"
      }
    ]
  },
  "status": 400
}

Use from/size for: User-facing search results (first ~20 pages).

Strategy 2: search_after (Keyset Pagination)

search_after uses the sort values from the last document to fetch the next page. It’s efficient and stateless.

Step 1: First page (same as normal search):

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "sort": [
    { "@timestamp": "desc" },
    { "_id": "asc" }
  ],
  "size": 3
}'

{
  "hits": {
    "hits": [
      { "_id": "doc6", "sort": [1743498000000, "doc6"], "_source": { "@timestamp": "2026-04-01T09:00:00Z" } },
      { "_id": "doc5", "sort": [1743495600000, "doc5"], "_source": { "@timestamp": "2026-04-01T08:20:00Z" } },
      { "_id": "doc4", "sort": [1743495300000, "doc4"], "_source": { "@timestamp": "2026-04-01T08:15:00Z" } }
    ]
  }
}

Step 2: Next page — pass the last hit’s sort values:

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "sort": [
    { "@timestamp": "desc" },
    { "_id": "asc" }
  ],
  "size": 3,
  "search_after": [1743495300000, "doc4"]
}'

{
  "hits": {
    "hits": [
      { "_id": "doc3", "sort": [1743495000000, "doc3"], "_source": { "@timestamp": "2026-04-01T08:10:00Z" } },
      { "_id": "doc2", "sort": [1743494700000, "doc2"], "_source": { "@timestamp": "2026-04-01T08:05:00Z" } },
      { "_id": "doc1", "sort": [1743494400000, "doc1"], "_source": { "@timestamp": "2026-04-01T08:00:00Z" } }
    ]
  }
}

Tip: Always include a tiebreaker field (like _id) in your sort to ensure unique ordering.

Use search_after for: Infinite scroll UIs, paginating beyond 10,000 results.

Strategy 3: Point in Time (PIT) API

PIT provides a consistent snapshot of the data. Combined with search_after, it ensures results don’t shift as new documents are indexed.

Step 1: Open a PIT:

curl -s -X POST "localhost:9200/web-logs/_pit?keep_alive=5m&pretty"

{
  "id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKwZub2RlXzIAAAAAAAAAAAwBYgACBHR5cGUFdXVpZA=="
}

Step 2: Search using the PIT (no index in the URL):

curl -s -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d '{
  "size": 3,
  "query": { "match_all": {} },
  "pit": {
    "id": "46ToAwMDaWR5BXV...",
    "keep_alive": "5m"
  },
  "sort": [
    { "@timestamp": "desc" }
  ]
}'

Step 3: Continue with search_after + PIT for subsequent pages.

Step 4: Close the PIT when done:

curl -s -X DELETE "localhost:9200/_pit" -H 'Content-Type: application/json' -d '{
  "id": "46ToAwMDaWR5BXV..."
}'

{
  "succeeded": true,
  "num_freed": 1
}

Use PIT for: Export jobs, reporting, any scenario needing consistent pagination.

Scroll API (Deprecated)

The Scroll API was the old way to iterate over large result sets. It’s now deprecated in favor of search_after + PIT. You may still see it in legacy code:

# Deprecated — use search_after + PIT instead
curl -s -X GET "localhost:9200/web-logs/_search?scroll=1m&pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "size": 100
}'

Note: The Scroll API holds a search context on the server, consuming resources. Prefer search_after + PIT for new implementations.

Pagination Strategy Comparison

Feature from/size search_after PIT + search_after
Max results 10,000 Unlimited Unlimited
Stateless Yes Yes No (PIT ID)
Consistent snapshot No No Yes
Random page access Yes No (sequential) No (sequential)
Best for UI pages 1-20 Infinite scroll Exports, reports

Source Filtering

Control which fields are returned in _source to reduce response size.

Include/Exclude Specific Fields

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "_source": ["method", "status_code", "response_time"],
  "size": 3
}'

{
  "hits": {
    "hits": [
      { "_source": { "method": "GET", "status_code": 200, "response_time": 120.5 } },
      { "_source": { "method": "POST", "status_code": 201, "response_time": 350.2 } },
      { "_source": { "method": "GET", "status_code": 200, "response_time": 95.3 } }
    ]
  }
}

Include/Exclude with Patterns

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "_source": {
    "includes": ["method", "status_*", "response_*"],
    "excludes": ["user_agent"]
  },
  "size": 2
}'

{
  "hits": {
    "hits": [
      { "_source": { "method": "GET", "status_code": 200, "response_time": 120.5 } },
      { "_source": { "method": "POST", "status_code": 201, "response_time": 350.2 } }
    ]
  }
}

Disable _source Entirely

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "_source": false,
  "size": 2
}'

{
  "hits": {
    "hits": [
      { "_index": "web-logs", "_id": "abc123", "_score": 1.0 },
      { "_index": "web-logs", "_id": "def456", "_score": 1.0 }
    ]
  }
}

The fields Parameter

The fields parameter is the modern alternative to _source filtering. It retrieves values from the document’s stored fields or from the mapping, applying format conversions:

curl -s -X GET "localhost:9200/web-logs/_search?pretty" -H 'Content-Type: application/json' -d '{
  "query": { "match_all": {} },
  "fields": ["method", "status_code", "@timestamp"],
  "_source": false,
  "size": 2
}'

{
  "hits": {
    "hits": [
      {
        "_id": "abc123",
        "fields": {
          "method": ["GET"],
          "status_code": [200],
          "@timestamp": ["2026-04-01T08:00:00.000Z"]
        }
      },
      {
        "_id": "def456",
        "fields": {
          "method": ["POST"],
          "status_code": [201],
          "@timestamp": ["2026-04-01T08:05:00.000Z"]
        }
      }
    ]
  }
}

Note: fields always returns values as arrays, even for single-valued fields.

Common Mistakes

Mistake 1: Deep Pagination with from/size

Using from: 50000, size: 10 to export data. This requires each shard to return 50,010 results.

Fix: Use search_after + PIT for exporting large datasets.

Mistake 2: Not Including a Tiebreaker in search_after

If two documents share the same sort value, search_after may skip or duplicate results.

Fix: Always add _id (or another unique field) as the last sort criterion.

Mistake 3: Forgetting to Close PIT

Open PIT contexts consume cluster resources. If you forget to close them, they expire after keep_alive but waste resources until then.

Fix: Always close PIT in a finally block in your application code.

Quick Reference

Feature Syntax Notes
Sort ascending { "field": "asc" } Default if direction omitted
Sort descending { "field": "desc" }  
Sort by score { "_score": "desc" } Include with other sort fields
Missing values { "field": { "order": "asc", "missing": "_last" } } _first, _last, or value
from/size "from": 0, "size": 10 Max from + size = 10000
search_after "search_after": [sort_values] Requires consistent sort order
PIT POST /index/_pit?keep_alive=5m Consistent snapshot for pagination
_source include "_source": ["field1", "field2"] Array or include/exclude object
_source disable "_source": false Only metadata returned
fields "fields": ["field1", "field2"] Modern alternative, returns arrays

Next: Practice Exercises — Test your querying skills with hands-on exercises.

You may also enjoy