ELK Stack — JVM and System Tuning

4 minute read

JVM and System Tuning

Elasticsearch runs on the JVM, so garbage collection behavior, heap sizing, and OS-level settings directly affect cluster stability and performance. Getting these right prevents out-of-memory crashes, long GC pauses, and degraded throughput.

Heap Size

The 50% / 31 GB Rules

  1. Set heap to 50% of physical RAM — the other 50% is for the OS filesystem cache, which Lucene relies on heavily
  2. Never exceed 31 GB — above ~31 GB, the JVM cannot use compressed ordinary object pointers (compressed oops), wasting memory
Node RAM Recommended Heap Filesystem Cache
16 GB 8 GB 8 GB
32 GB 16 GB 16 GB
64 GB 31 GB 33 GB
128 GB 31 GB 97 GB

Warning: Setting heap above 31 GB is almost always worse than 31 GB because you lose compressed oops, effectively wasting ~4 GB of the increase.

Setting Heap Size

Edit the JVM options file:

# /etc/elasticsearch/jvm.options.d/heap.options
-Xms16g
-Xmx16g

Important: Always set -Xms and -Xmx to the same value. Different values cause the JVM to pause while resizing the heap.

Verify Heap Settings

curl -s localhost:9200/_nodes/stats/jvm?filter_path=nodes.*.jvm.mem | python3 -m json.tool

{
  "nodes": {
    "abc123": {
      "jvm": {
        "mem": {
          "heap_used_in_bytes": 8589934592,
          "heap_used_percent": 52,
          "heap_committed_in_bytes": 17179869184,
          "heap_max_in_bytes": 17179869184
        }
      }
    }
  }
}

Garbage Collection

G1GC (Default in ES 8.x)

Elasticsearch 8.x ships with G1GC as the default garbage collector. G1GC divides the heap into regions and collects the regions with the most garbage first.

# /etc/elasticsearch/jvm.options.d/gc.options

# G1GC is default — these are tuning knobs
-XX:+UseG1GC
-XX:G1HeapRegionSize=16m
-XX:InitiatingHeapOccupancyPercent=40
-XX:G1ReservePercent=15
-XX:MaxGCPauseMillis=200
Parameter Default Purpose
G1HeapRegionSize Auto Size of each G1 region (1–32 MB)
InitiatingHeapOccupancyPercent 45 Start concurrent GC when heap is this full
G1ReservePercent 10 Reserve heap to reduce promotion failures
MaxGCPauseMillis 200 Target maximum GC pause time

GC Logging

Enable GC logs to diagnose performance issues:

# /etc/elasticsearch/jvm.options.d/gc-logging.options
-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

Interpreting GC Logs

Watch for:

  • Long pauses (> 500 ms) — indicates heap pressure or too many old-gen objects
  • Frequent full GCs — heap is too small or there is a memory leak
  • To-space exhaustion — G1 cannot find empty regions; increase G1ReservePercent

Memory Pressure

Monitor JVM memory pressure via the nodes stats API:

curl -s "localhost:9200/_nodes/stats/jvm?filter_path=nodes.*.jvm.mem.heap_used_percent"
Heap Used % Status Action
< 75% Healthy None
75–85% Warning Investigate queries, reduce shard count
85–95% Critical Add nodes, increase heap, enable circuit breakers
> 95% Danger Node may OOM; fix immediately

Circuit Breakers

Circuit breakers prevent operations from consuming too much memory and crashing the node.

Breaker Default Limit Purpose
indices.breaker.fielddata.limit 40% of heap Field data (text field aggregations)
indices.breaker.request.limit 60% of heap Per-request data structures
indices.breaker.total.limit 70% of heap Combined limit across all breakers
network.breaker.inflight_requests.limit 100% of heap In-flight network data

Configuring Circuit Breakers

curl -X PUT "localhost:9200/_cluster/settings" \
  -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "indices.breaker.fielddata.limit": "35%",
    "indices.breaker.request.limit": "55%",
    "indices.breaker.total.limit": "65%"
  }
}'

When Breakers Trip

You will see errors like:

{
  "error": {
    "type": "circuit_breaking_exception",
    "reason": "[fielddata] Data too large, data for [user.keyword] would be [11512345600/10.7gb], which is larger than the limit of [10737418240/10gb]"
  },
  "status": 429
}

Fixes:

  • Reduce fielddata usage — avoid text field aggregations
  • Use keyword fields or doc_values for aggregations
  • Increase heap (if under 31 GB)
  • Add more data nodes to spread the load

OS-Level Settings

The operating system must be configured to support Elasticsearch’s requirements.

Disable Swapping

Swapping destroys Elasticsearch performance. Disable it completely:

# Option 1: Disable swap entirely (recommended)
sudo swapoff -a
# Remove swap entries from /etc/fstab

# Option 2: Set swappiness to 1
sudo sysctl vm.swappiness=1
echo "vm.swappiness=1" | sudo tee -a /etc/sysctl.conf

Or use memory locking in Elasticsearch:

# elasticsearch.yml
bootstrap.memory_lock: true

# Verify memory lock
curl -s localhost:9200/_nodes?filter_path=**.mlockall

{
  "nodes": {
    "abc123": {
      "process": {
        "mlockall": true
      }
    }
  }
}

File Descriptors

Each shard opens many files. Set a high file descriptor limit:

# /etc/security/limits.conf
elasticsearch  -  nofile  65535

# Verify
curl -s localhost:9200/_nodes/stats/process?filter_path=**.max_file_descriptors

Virtual Memory

Lucene uses mmap to map segment files. Increase the vm.max_map_count:

sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf

# Verify
sysctl vm.max_map_count

Thread Count

Ensure sufficient threads:

# /etc/security/limits.conf
elasticsearch  -  nproc  4096

Complete jvm.options Example

# /etc/elasticsearch/jvm.options.d/custom.options

## Heap — 50% of 64 GB RAM
-Xms31g
-Xmx31g

## GC — G1GC tuning
-XX:+UseG1GC
-XX:G1HeapRegionSize=16m
-XX:InitiatingHeapOccupancyPercent=40
-XX:G1ReservePercent=15
-XX:MaxGCPauseMillis=200

## GC logging
-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

## Performance
-XX:+AlwaysPreTouch
-server

## Error handling
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

Monitoring JVM Health

# JVM stats for all nodes
curl -s localhost:9200/_nodes/stats/jvm?pretty

# Specific node
curl -s localhost:9200/_nodes/hot-01/stats/jvm?pretty

# Compact view — heap percent per node
curl -s localhost:9200/_cat/nodes?v&h=name,heap.percent,heap.current,heap.max,ram.percent,cpu

name    heap.percent heap.current heap.max ram.percent cpu
hot-01            52       16.1gb   31gb           78  12
hot-02            48       14.9gb   31gb           75  10
warm-01           35        5.4gb   16gb           45   4

Quick Reference

Setting Recommended Value
Heap size 50% of RAM, max 31 GB
-Xms = -Xmx Always equal
GC G1GC (default in ES 8.x)
bootstrap.memory_lock true
vm.swappiness 0 or 1
vm.max_map_count 262144
nofile 65535
nproc 4096
Fielddata breaker 35–40% of heap
Total breaker 65–70% of heap

← Shard Strategy · Next → Indexing Performance

You may also enjoy