ELK Stack — Indexing Performance

5 minute read

Indexing Performance

Indexing is the process of writing documents into Elasticsearch. In high-throughput environments — log ingestion, metrics collection, ETL pipelines — indexing speed is the primary bottleneck. This section covers every lever you can pull to maximize write performance.

The Bulk API

Never index documents one at a time. The bulk API amortizes the overhead of HTTP connections, request parsing, and shard routing across many documents in a single request.

curl -X POST "localhost:9200/_bulk" \
  -H 'Content-Type: application/x-ndjson' -d'
{"index": {"_index": "logs-2026.04", "_id": "1"}}
{"@timestamp": "2026-04-01T10:00:00Z", "level": "INFO", "message": "App started"}
{"index": {"_index": "logs-2026.04", "_id": "2"}}
{"@timestamp": "2026-04-01T10:00:01Z", "level": "ERROR", "message": "Connection refused"}
{"index": {"_index": "logs-2026.04", "_id": "3"}}
{"@timestamp": "2026-04-01T10:00:02Z", "level": "WARN", "message": "Slow query detected"}
'

Bulk Sizing

Bulk Size	Effect
< 1 MB	Too small — HTTP overhead dominates
5–15 MB	Ideal range for most clusters
15–50 MB	Can work on beefy nodes, test carefully
> 50 MB	Risk of request timeouts and memory pressure

Tip: Measure in megabytes, not document count. 1000 documents at 1 KB each = 1 MB. 1000 documents at 50 KB each = 50 MB. Always size by payload.

Optimal Bulk Workers

Run multiple bulk requests in parallel from the client side:

Data Nodes	Recommended Bulk Threads
3	3–6
6	6–12
12	12–24

Start low and increase until you see 429 (too many requests) or CPU saturation.

Refresh Interval

By default, Elasticsearch refreshes every index every 1 second, making new documents searchable. During bulk loads, this is wasteful.

Increase During Bulk Loads

# Set refresh to 30 seconds during bulk ingestion
curl -X PUT "localhost:9200/logs-2026.04/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.refresh_interval": "30s"
}'

Disable Completely During Initial Load

# Disable refresh entirely
curl -X PUT "localhost:9200/logs-2026.04/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.refresh_interval": "-1"
}'

# ... perform bulk load ...

# Re-enable and force a refresh
curl -X PUT "localhost:9200/logs-2026.04/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.refresh_interval": "1s"
}'

curl -X POST "localhost:9200/logs-2026.04/_refresh"

Replica Management During Load

Each replica doubles the indexing work. For initial data loads, temporarily set replicas to zero:

# Set replicas to 0
curl -X PUT "localhost:9200/logs-2026.04/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.number_of_replicas": 0
}'

# ... perform bulk load ...

# Restore replicas
curl -X PUT "localhost:9200/logs-2026.04/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.number_of_replicas": 1
}'

Warning: With 0 replicas, you have no fault tolerance. Only do this during initial bulk loads when the source data is still available for re-ingestion.

Translog Settings

The transaction log (translog) provides crash recovery. Every write goes to the translog before being applied to the Lucene index. By default, Elasticsearch fsyncs the translog after every request.

Async Translog for Higher Throughput

curl -X PUT "localhost:9200/logs-2026.04/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.translog.durability": "async",
  "index.translog.sync_interval": "30s",
  "index.translog.flush_threshold_size": "1gb"
}'

Setting	Default	Tuned
`durability`	`request`	`async`
`sync_interval`	`5s`	`30s`
`flush_threshold_size`	`512mb`	`1gb`

Warning: Async translog means you can lose up to sync_interval worth of data on a crash. Acceptable for logs, not for financial transactions.

Index Codec

Elasticsearch uses LZ4 compression by default for a balance of speed and compression. For warm/cold data where write speed is less important:

curl -X PUT "localhost:9200/logs-archive/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.codec": "best_compression"
}'

Codec	Compression	Speed
`default` (LZ4)	Good	Fast
`best_compression` (DEFLATE)	Better (~30% smaller)	Slower writes

Note: Changing codec only affects new segments. Existing data keeps its original codec until a force merge.

Merge Policy

Lucene continuously merges small segments into larger ones. Tuning the merge policy controls how aggressively merging happens:

curl -X PUT "localhost:9200/logs-2026.04/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.merge.policy.max_merged_segment": "5gb",
  "index.merge.policy.segments_per_tier": 10,
  "index.merge.policy.max_merge_at_once": 10,
  "index.merge.scheduler.max_thread_count": 1
}'

Setting	Default	Purpose
`max_merged_segment`	5 GB	Largest segment the merge policy will create
`segments_per_tier`	10	How many segments per tier before merging
`max_merge_at_once`	10	Max segments to merge in one operation
`max_thread_count`	`max(1, cores/2)`	Merge threads — set to 1 for spinning disks

Indexing Buffer

The indexing buffer holds documents in memory before they are written to a Lucene segment. The default is 10% of heap, shared across all shards on the node.

# elasticsearch.yml
indices.memory.index_buffer_size: 15%
indices.memory.min_index_buffer_size: 96mb

Increase the buffer if you have high indexing throughput and relatively few shards per node.

Force Merge

After an index is no longer being written to, force merge it down to a single segment per shard. This reduces file handles, heap usage, and improves search speed.

# Force merge to 1 segment per shard
curl -X POST "localhost:9200/logs-2026.03/_forcemerge?max_num_segments=1"

Warning: Force merge is I/O intensive and temporarily doubles disk usage for the index. Never run it on an index that is still receiving writes.

Automate via ILM

{
  "warm": {
    "min_age": "7d",
    "actions": {
      "forcemerge": {
        "max_num_segments": 1
      }
    }
  }
}

Data Streams

Data streams are the recommended way to manage time-series data. They automatically handle index creation, rollover, and ILM.

# Create an index template for a data stream
curl -X PUT "localhost:9200/_index_template/logs-template" \
  -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logs-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.refresh_interval": "5s",
      "index.lifecycle.name": "logs-lifecycle"
    }
  }
}'

# Create the data stream
curl -X PUT "localhost:9200/_data_stream/logs-myapp"

# Index into the data stream (must use op_type=create)
curl -X POST "localhost:9200/logs-myapp/_doc" \
  -H 'Content-Type: application/json' -d'
{
  "@timestamp": "2026-04-01T10:00:00Z",
  "level": "INFO",
  "message": "Service started"
}'

Benchmarking with ES Rally

ES Rally is the official benchmarking tool for Elasticsearch. Always benchmark before and after tuning changes.

# Install ES Rally
pip install esrally

# Run the geonames benchmark against your cluster
esrally race --track=geonames --target-hosts=localhost:9200 --pipeline=benchmark-only

# Compare two runs
esrally compare --baseline=<run-id-1> --contender=<run-id-2>

Key metrics to watch:

Metric	What It Measures
`Median Throughput`	Docs/sec during indexing
`Service Time (p99)`	99th percentile latency
`Error Rate`	Percentage of failed requests
`Merge Time`	Time spent merging segments
`Refresh Time`	Time spent refreshing

Indexing Optimization Checklist

Step	Action	Expected Impact
1	Use bulk API (5–15 MB payloads)	10–50x throughput increase
2	Increase `refresh_interval` to 30s	10–30% throughput increase
3	Set replicas to 0 during initial load	~2x throughput increase
4	Use async translog	10–20% throughput increase
5	Auto-generate `_id` (omit `_id` in bulk)	10–15% throughput increase
6	Size bulk workers to match data nodes	Saturate cluster capacity
7	Force merge after write completes	Better search performance

Quick Reference

Setting	Recommended Value
Bulk payload size	5–15 MB
`refresh_interval` (ingestion)	30s or `-1`
`number_of_replicas` (initial load)	0
`translog.durability` (logs)	`async`
`translog.sync_interval`	30s
`index.codec` (warm/cold)	`best_compression`
`merge.scheduler.max_thread_count` (HDD)	1
`index_buffer_size`	10–15% of heap
Force merge target	1 segment per shard

← JVM and System Tuning · Next → Query Optimization

Share on

Twitter Facebook LinkedIn