ELK Stack — Capacity Planning

5 minute read

Capacity Planning for Elasticsearch

Capacity planning determines how much storage, RAM, CPU, and network bandwidth your cluster needs — today and as it grows. Under-provisioning leads to outages; over-provisioning wastes money. This section provides formulas, hardware profiles, and reference architectures to help you size your cluster correctly.

Storage Estimation

The core formula:

Total storage = Daily ingest × Retention days × (1 + Replicas) × Overhead
Factor Typical Value
Overhead (indexing, segment merges, OS reserved) 1.10–1.15× (10–15%)
Replicas 1 (doubles raw data)

Example — Log Analytics

Parameter Value
Daily raw ingest 100 GB
Retention 30 days
Replicas 1
Overhead 15% 
Total = 100 GB × 30 × (1 + 1) × 1.15
     = 100 × 30 × 2 × 1.15
     = 6,900 GB  ≈  6.9 TB

Compression Considerations

Elasticsearch uses best_speed (LZ4) compression by default. You can switch to best_compression (DEFLATE) for cold data:

curl -X PUT "localhost:9200/logs-cold/_settings" -H 'Content-Type: application/json' -d'
{
  "index.codec": "best_compression"
}'

This can reduce storage by 15–25% but increases CPU during merges.

RAM Requirements

The golden rule:

JVM heap = 50% of available RAM, but never more than 31 GB.

The other 50% is for the OS filesystem cache, which Elasticsearch relies on heavily for Lucene segment reads.

Node RAM Heap (-Xmx) OS Cache
16 GB 8 GB 8 GB
32 GB 16 GB 16 GB
64 GB 31 GB 33 GB
128 GB 31 GB 97 GB

Important: Do not set heap above 31 GB. Beyond this threshold, the JVM cannot use compressed ordinary object pointers (compressed oops), which dramatically increases memory usage.

Set heap in jvm.options:

-Xms16g
-Xmx16g

CPU Sizing

CPU requirements depend on:

Workload CPU Profile
Heavy indexing (logs, metrics) More cores (8–16 per node)
Heavy search (analytics, dashboards) More cores + fast single-thread
Heavy aggregations More cores
Ingest pipelines (grok, geoip) More cores

Rule of thumb: Start with 8 vCPUs per data node for moderate workloads. Monitor thread pool rejections and scale up if needed.

Network Considerations

  • Intra-cluster (transport): 10 Gbps minimum between nodes. Shard recovery and replication are network-intensive.
  • Client ingestion: Depends on ingest rate. 100 GB/day ≈ ~10 Mbps sustained.
  • Cross-AZ traffic: Factor in cloud provider cross-AZ data transfer costs.

Hot / Warm / Cold Hardware Profiles

Use ILM to move data through tiers as it ages:

Tier Hardware Use Case
Hot NVMe SSD, 64 GB RAM, 16 cores Active indexing + recent searches
Warm SAS SSD or fast HDD, 64 GB RAM, 8 cores Read-only, less frequent queries
Cold Large HDD, 32 GB RAM, 4 cores Rarely accessed, compliance/archive
Frozen Searchable snapshots (S3/GCS) On-demand access, minimal local storage

Node configuration:

# Hot node
node.roles: [data_hot, ingest]
node.attr.data: hot

# Warm node
node.roles: [data_warm]
node.attr.data: warm

# Cold node
node.roles: [data_cold]
node.attr.data: cold

# Frozen node
node.roles: [data_frozen]

Reference Architectures

Small — Development / Low-Volume Production

Component Spec Count
Master + Data 4 vCPU, 16 GB RAM, 500 GB SSD 3
Kibana 2 vCPU, 4 GB RAM 1
Logstash 4 vCPU, 8 GB RAM 1
Daily ingest Up to 50 GB/day  

Medium — General Production

Component Spec Count
Dedicated Master 2 vCPU, 8 GB RAM, 50 GB SSD 3
Hot Data 16 vCPU, 64 GB RAM, 2 TB NVMe 3
Warm Data 8 vCPU, 64 GB RAM, 8 TB SAS SSD 3
Coordinating 8 vCPU, 16 GB RAM 2
Kibana 4 vCPU, 8 GB RAM 2
Logstash 8 vCPU, 16 GB RAM 2
Daily ingest 50–500 GB/day  

Large — High-Volume / Enterprise

Component Spec Count
Dedicated Master 4 vCPU, 16 GB RAM, 100 GB SSD 3
Hot Data 32 vCPU, 128 GB RAM, 4 TB NVMe 6+
Warm Data 16 vCPU, 64 GB RAM, 16 TB HDD 6+
Cold Data 8 vCPU, 32 GB RAM, 32 TB HDD 4+
Coordinating 16 vCPU, 32 GB RAM 3+
Kibana 8 vCPU, 16 GB RAM 3
Logstash 16 vCPU, 32 GB RAM 4+
Daily ingest 500 GB – 5 TB/day  

Sizing for Common Use Cases

Log Analytics (100 GB/day, 30 days retention)

Storage  = 100 × 30 × 2 × 1.15 = 6.9 TB
Nodes    = 6.9 TB / 2 TB per hot node ≈ 4 hot nodes
           (plus warm nodes if retention > 7 days hot)
RAM      = 4 × 64 GB = 256 GB total

APM (50 GB/day, 14 days retention)

Storage  = 50 × 14 × 2 × 1.15 = 1.61 TB
Nodes    = 2 hot nodes (2 TB each)

SIEM (200 GB/day, 90 days retention)

Storage  = 200 × 90 × 2 × 1.15 = 41.4 TB
Hot (7d) = 200 × 7 × 2 × 1.15 = 3.22 TB → 2 hot nodes
Warm (83d) = 200 × 83 × 2 × 1.15 = 38.18 TB → 5 warm nodes (8 TB each)

Growth Planning

Plan for at least 6–12 months of data growth:

  1. Measure current daily ingest — Use _cat/indices?v&s=creation.date:desc to check recent index sizes.
  2. Project growth rate — Are new data sources being added? Is log verbosity increasing?
  3. Add headroom — Size for 1.3× projected peak (30% buffer).
  4. Review quarterly — Adjust node count and hardware tier allocation.

Benchmarking with esrally

esrally is the official Elasticsearch benchmarking tool.

Install

pip install esrally

Run a Benchmark

esrally race --track=geonames --target-hosts=localhost:9200 --pipeline=benchmark-only

Common Tracks

Track Description
geonames 11M docs, mixed workload
http_logs 247M docs, log analytics
pmc 574K docs, full-text search
nyc_taxis 165M docs, heavy aggregations

Compare results across hardware configurations to find the best price-performance ratio.

Cloud vs On-Prem Considerations

Factor Cloud (Elastic Cloud / AWS OpenSearch) On-Prem
Scaling Instant (add nodes in minutes) Weeks (procurement)
Cost model OpEx (pay-as-you-go) CapEx (upfront hardware)
Maintenance Managed (upgrades, patches) Self-managed
Customization Limited (preset instance types) Full control
Data residency Choose region Full control
Cost at scale Can be expensive Often cheaper at high volume

Quick Reference

Metric Formula / Guideline
Storage daily_ingest × retention × (1 + replicas) × 1.15
JVM Heap 50% of RAM, max 31 GB
Shards per node < 1,000
Shard size 10–50 GB (ideal)
Shards per GB heap < 20
Refresh interval (heavy ingest) 30s–60s
Thread pool write queue Monitor for rejections

Next Steps

With your cluster sized correctly, you need a plan for keeping it up to date. Continue to Upgrade Strategies.

You may also enjoy