ELK Stack — Shard Strategy

5 minute read

Shard Strategy

Every Elasticsearch index is split into shards — the fundamental unit of storage, search, and replication. Getting the shard count and size right is one of the most impactful performance decisions you will make.

How Shards Work

Each shard is a self-contained Lucene index. When you index a document, Elasticsearch hashes its _id (or a custom routing value) to pick one primary shard. That shard processes the write and forwards it to its replica shards.

When you search, Elasticsearch fans the query out to one copy of every shard (primary or replica), each shard returns local results, and a coordinating node merges them.

Shard Sizing Guidelines

Target: 10–50 GB Per Shard

Shard Size Impact
< 1 GB Too small — excessive overhead, bloated cluster state
1–10 GB Acceptable for small indices, but watch total count
10–50 GB Ideal range for most workloads
50–100 GB Acceptable if queries are simple; recovery takes longer
> 100 GB Recovery and rebalancing become very slow

Sizing Formula

number_of_primary_shards = ceil(expected_index_size_GB / target_shard_size_GB)

Example: You expect a daily log index to reach 120 GB.

primary_shards = ceil(120 / 40) = 3

With 1 replica, total shards = 3 primary + 3 replica = 6.

Shards Per Node Limits

Rule of Thumb: ~20 Shards Per GB of Heap

A node with 30 GB heap should hold no more than ~600 shards. This is a soft guideline — fewer shards is almost always better.

# Check shards per node
curl -s localhost:9200/_cat/allocation?v

shards disk.indices disk.used disk.avail disk.total disk.percent host        node
  142       1.2tb     1.4tb      2.6tb      4tb           35   10.0.1.10   hot-01
  138       1.1tb     1.3tb      2.7tb      4tb           32   10.0.1.11   hot-02

Hard Limit

You can enforce a per-node cap:

# elasticsearch.yml
cluster.max_shards_per_node: 1000

Or via the API:

curl -X PUT "localhost:9200/_cluster/settings" \
  -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.max_shards_per_node": 1000
  }
}'

Over-Sharding Problems

Over-sharding happens when you have too many small shards. Symptoms:

  1. High cluster state overhead — every shard adds metadata to the cluster state
  2. Excessive file handles — each shard opens multiple Lucene segment files
  3. Slow master operations — shard allocation decisions take longer
  4. Poor search performance — coordinating node merges thousands of partial results
  5. Memory pressure — each shard consumes heap for segment metadata, field data caches

Common Causes

  • Default 1-shard-per-index with thousands of tiny indices
  • Daily indices that never grow beyond a few hundred megabytes
  • Using index-per-tenant without consolidation

How to Fix

  • Use data streams with rollover based on size (not just time)
  • Shrink old indices: POST /my-index/_shrink/my-index-shrunk
  • Merge small daily indices into weekly or monthly indices
  • Increase index.number_of_shards for new indices that will be large

Under-Sharding Problems

Under-sharding happens when you have too few shards for a large index:

  1. Uneven distribution — fewer shards than data nodes means some nodes sit idle
  2. Large shard recovery — a 200 GB shard takes much longer to recover than four 50 GB shards
  3. Indexing bottleneck — writes are limited by the number of primary shards
  4. Search hotspots — all search traffic hits a small number of nodes

How to Fix

  • Reindex with more primary shards
  • Use rollover with max_primary_shard_size to auto-split
  • Split API for emergency increases (doubles shard count)
# Split an index from 1 to 2 primary shards
curl -X POST "localhost:9200/my-index/_split/my-index-split" \
  -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.number_of_shards": 2
  }
}'

Allocation Awareness

Allocation awareness ensures Elasticsearch spreads replicas across failure domains (racks, zones, data centers).

Rack / Zone Awareness

# elasticsearch.yml — each node declares its zone
node.attr.zone: us-east-1a

# On any node or via API — tell ES to be aware of zones
cluster.routing.allocation.awareness.attributes: zone

With this setting, Elasticsearch ensures that a primary shard and its replica are never on nodes in the same zone.

Forced Awareness

Forced awareness prevents shard allocation when a zone is missing. This avoids overloading the surviving zone:

curl -X PUT "localhost:9200/_cluster/settings" \
  -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone",
    "cluster.routing.allocation.awareness.force.zone.values": "us-east-1a,us-east-1b,us-east-1c"
  }
}'

If zone us-east-1b goes down, replicas that belonged there remain unassigned rather than piling onto us-east-1a or us-east-1c.

Allocation Filtering

Move specific indices to specific nodes using allocation filters:

# Move index to warm nodes only
curl -X PUT "localhost:9200/logs-2026.03/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.require.box_type": "warm"
}'
Filter Meaning
require Shard MUST be on a node matching the attribute
include Shard CAN be on a node matching the attribute
exclude Shard MUST NOT be on a node matching the attribute

Shard Routing

By default, documents are routed to shards based on a hash of _id. Custom routing sends related documents to the same shard, improving search speed for queries that include the routing value.

# Index with custom routing
curl -X PUT "localhost:9200/orders/_doc/1001?routing=customer_42" \
  -H 'Content-Type: application/json' -d'
{
  "customer_id": "customer_42",
  "amount": 250.00
}'

# Search with routing — hits only one shard
curl -X GET "localhost:9200/orders/_search?routing=customer_42" \
  -H 'Content-Type: application/json' -d'
{
  "query": { "term": { "customer_id": "customer_42" } }
}'

Warning: Custom routing can create hot shards if the distribution is uneven. Always analyze your routing key cardinality before enabling.

Shard Sizing Cheat Sheet

Scenario Primary Shards Target Shard Size Replicas
Small index (< 10 GB) 1 — 1
Medium index (10–100 GB) 2–5 20–50 GB 1
Large daily logs (500 GB/day) 10–15 30–50 GB 1
Search-heavy (low volume) 1–3 10–30 GB 1–2
Analytics (large aggs) 3–5 30–50 GB 1

Common Mistakes

Mistake Why It Hurts Fix
5 shards on a 500 MB index 100 MB per shard — massive overhead Use 1 shard
1 shard on a 500 GB index Slow recovery, search bottleneck Use 10–15 shards
No replicas Zero fault tolerance Set replicas to at least 1
Ignoring allocation awareness Both copies on same rack Configure zone awareness
Never shrinking old indices Thousands of tiny shards accumulate Shrink + force-merge via ILM

Monitoring Shards

# Shard distribution
curl -s localhost:9200/_cat/shards?v&s=index

# Shard sizes
curl -s localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,store&s=store:desc

# Indices with shard counts
curl -s localhost:9200/_cat/indices?v&h=index,pri,rep,docs.count,store.size&s=store.size:desc

Quick Reference

Setting Recommended Value
Shard size 10–50 GB
Shards per GB heap ~20 (soft limit)
cluster.max_shards_per_node 1000 (adjust per workload)
Allocation awareness Always in multi-zone clusters
Forced awareness Enable to prevent zone overload
Custom routing Use when queries always filter on a key
Replicas At least 1 for production

← Cluster Architecture · Next → JVM and System Tuning

You may also enjoy