ELK Stack — Cluster Monitoring

4 minute read

Cluster Monitoring

Monitoring your ELK Stack is essential for detecting issues before they escalate into outages. Elasticsearch exposes a rich set of APIs for cluster health, node statistics, and shard status. Combined with Stack Monitoring in Kibana and external tools like Prometheus, you get full visibility into every component.

Cluster Health API

The simplest starting point — returns a single status for the entire cluster:

curl -s "localhost:9200/_cluster/health" | jq .

{
  "cluster_name": "production-elk",
  "status": "green",
  "number_of_nodes": 9,
  "number_of_data_nodes": 6,
  "active_primary_shards": 150,
  "active_shards": 300,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100.0
}
Status Meaning
green All primary and replica shards are assigned
yellow All primaries assigned, some replicas unassigned
red One or more primary shards are unassigned — data loss risk

Wait for a specific status (useful in scripts):

curl -s "localhost:9200/_cluster/health?wait_for_status=green&timeout=60s"

Cluster Stats

Aggregated statistics across the entire cluster:

curl -s "localhost:9200/_cluster/stats" | jq '.nodes, .indices'

Key fields to monitor:

  • indices.count — Total number of indices
  • indices.shards.total — Total shard count (keep under ~1,000 per node)
  • nodes.jvm.mem.heap_used_in_bytes — Aggregate heap usage
  • nodes.fs.total_in_bytes / nodes.fs.available_in_bytes — Disk capacity

Node Stats

Per-node breakdown of JVM, OS, thread pools, indexing, and search:

curl -s "localhost:9200/_nodes/stats" | jq '.nodes | to_entries[0].value | keys'

Key Metrics by Category

Category Metric Path What to Watch
JVM Heap jvm.mem.heap_used_percent Stay below 75%
GC jvm.gc.collectors.old.collection_time_in_millis Spikes indicate memory pressure
Indexing indices.indexing.index_total Rate of documents indexed
Search indices.search.query_time_in_millis Average search latency
Thread Pools thread_pool.write.rejected Rejections mean overload
Circuit Breakers breakers.parent.tripped Trips mean queries are too large
Disk fs.total.available_in_bytes Alert below 15% free

Filtered Node Stats

Get only JVM and thread pool stats:

curl -s "localhost:9200/_nodes/stats/jvm,thread_pool" | jq .

_cat APIs

Human-readable tabular output, ideal for quick diagnosis:

Nodes Overview

curl -s "localhost:9200/_cat/nodes?v&h=name,ip,role,heap.percent,cpu,load_1m,disk.used_percent"

name         ip          role   heap.percent cpu load_1m disk.used_percent
es-master-1  10.0.1.10   m              22   3    0.45              32.1
es-data-1a   10.0.1.11   d              61  18    2.31              58.4
es-data-1b   10.0.1.12   d              55  14    1.87              52.7

Shard Allocation

curl -s "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,docs,store,node&s=index"

Index Sizes

curl -s "localhost:9200/_cat/indices?v&h=index,health,pri,rep,docs.count,store.size&s=store.size:desc"

Allocation Summary

curl -s "localhost:9200/_cat/allocation?v"

shards disk.indices disk.used disk.avail disk.total disk.percent host       node
    50       120gb    130gb      370gb      500gb           26 10.0.1.11 es-data-1a
    50       115gb    125gb      375gb      500gb           25 10.0.1.12 es-data-1b

Pending Tasks

curl -s "localhost:9200/_cat/pending_tasks?v"

A growing list of pending tasks indicates the master node is overloaded.

Stack Monitoring in Kibana

Kibana’s built-in Stack Monitoring provides dashboards for Elasticsearch, Logstash, Beats, and APM Server.

Self-monitoring is deprecated since ES 7.x. Use Metricbeat to ship monitoring data to a dedicated monitoring cluster (or the same cluster for small deployments).

Step 1 — Enable Monitoring Collection on ES

# elasticsearch.yml
xpack.monitoring.collection.enabled: false   # Disable self-collection
xpack.monitoring.elasticsearch.collection.enabled: false

Step 2 — Configure Metricbeat on Each ES Node

# metricbeat.yml
metricbeat.modules:
  - module: elasticsearch
    xpack.enabled: true
    period: 10s
    hosts: ["https://localhost:9200"]
    username: "monitoring_user"
    password: "${ES_MONITORING_PASSWORD}"
    ssl.verification_mode: certificate

output.elasticsearch:
  hosts: ["https://monitoring-cluster:9200"]
  username: "metricbeat_writer"
  password: "${MB_PASSWORD}"

Step 3 — Open Stack Monitoring in Kibana

Navigate to Stack Monitoring → You should see cluster overview, node stats, index stats, and shard status.

Monitoring Logstash

Use Metricbeat with the logstash module:

metricbeat.modules:
  - module: logstash
    xpack.enabled: true
    period: 10s
    hosts: ["localhost:9600"]

Key Logstash metrics:

  • Events in/out per second — Pipeline throughput
  • Pipeline worker utilization — Are workers busy?
  • Queue backpressure — Persistent queue growing?

Monitoring Beats

Beats can self-report monitoring data:

# filebeat.yml
monitoring.enabled: true
monitoring.elasticsearch:
  hosts: ["https://monitoring-cluster:9200"]
  username: "beats_monitoring"
  password: "${BEATS_MON_PASSWORD}"

Prometheus Exporter for Elasticsearch

For teams that use Prometheus + Grafana instead of Stack Monitoring:

Deploy elasticsearch-exporter

docker run -d --name es-exporter \
  -p 9114:9114 \
  quay.io/prometheuscommunity/elasticsearch-exporter:latest \
  --es.uri=http://es-coord-1:9200 \
  --es.all \
  --es.indices \
  --es.shards

Prometheus Scrape Config

# prometheus.yml
scrape_configs:
  - job_name: 'elasticsearch'
    static_configs:
      - targets: ['es-exporter:9114']
    scrape_interval: 15s

Grafana Dashboard

Import the community dashboard ID 2322 or ID 14191 from grafana.com/dashboards.

Key Metrics to Watch — Summary

Metric Healthy Range Alert Threshold
Cluster status green yellow/red
JVM heap used % < 75% > 85%
GC old collection time Low, stable Spikes > 1s
Indexing rate Stable Sudden drop/spike
Search latency (p99) < 500ms > 2s
Thread pool write rejections 0 > 0
Circuit breaker trips 0 > 0
Disk used % < 80% > 85%
Pending cluster tasks 0 > 10 for > 1 min
Unassigned shards 0 > 0

Cluster Alerts

Set up Kibana alerting rules for critical conditions:

  1. Stack Monitoring → Alerts → Enable built-in rules:
    • Cluster health changed
    • Nodes changed
    • Disk usage threshold
    • Memory usage threshold
    • CPU usage threshold
  2. Actions — Send to Slack, PagerDuty, email, or webhook.
# Example: check health via script (cron fallback)
STATUS=$(curl -s localhost:9200/_cluster/health | jq -r '.status')
if [ "$STATUS" != "green" ]; then
  echo "ALERT: Cluster status is $STATUS" | mail -s "ES Alert" ops@example.com
fi

Quick Reference

Task Command
Cluster health GET _cluster/health
Cluster stats GET _cluster/stats
Node stats (all) GET _nodes/stats
Node stats (JVM) GET _nodes/stats/jvm
Cat nodes GET _cat/nodes?v
Cat shards GET _cat/shards?v
Cat indices GET _cat/indices?v&s=store.size:desc
Cat allocation GET _cat/allocation?v
Pending tasks GET _cat/pending_tasks?v
Thread pool stats GET _cat/thread_pool?v&h=name,active,rejected,completed

Next Steps

Now that you can observe your cluster, the next step is making sure it has enough resources. Continue to Capacity Planning.

You may also enjoy