ELK Stack — High Availability

4 minute read

High Availability for Elasticsearch

High Availability (HA) ensures your Elasticsearch cluster continues to serve reads and writes even when individual nodes, racks, or entire availability zones fail. A well-designed HA deployment combines dedicated master nodes, zone-aware shard allocation, replica strategies, and load balancing.

Multi-Node Cluster Architecture

A production cluster separates responsibilities across node roles:

Role Purpose Minimum Count
Master-eligible Cluster state management, shard allocation 3
Data Stores shards, executes searches and indexing 2+
Ingest Pre-processes documents (pipelines) 1+ (optional)
Coordinating-only Routes requests, scatter-gather 1+ (optional)
ML Machine learning jobs As needed

Dedicated Master Nodes

The master node is responsible for cluster-wide operations — creating/deleting indices, allocating shards, and tracking node membership. Never run data workloads on master nodes.

Why 3 Master-Eligible Nodes?

Elasticsearch uses a quorum-based consensus algorithm. With 3 master-eligible nodes the quorum is 2, meaning the cluster tolerates one master node failure:

Quorum = (number_of_master_eligible_nodes / 2) + 1
       = (3 / 2) + 1 = 2

Voting Configuration

Starting in ES 7.x, the voting configuration is managed automatically. You can inspect it:

curl -s localhost:9200/_cluster/state?filter_path=metadata.cluster_coordination.last_committed_config | jq .

For initial bootstrap, set cluster.initial_master_nodes only on the first start:

# elasticsearch.yml — first start only
cluster.initial_master_nodes:
  - es-master-1
  - es-master-2
  - es-master-3

Warning: Remove cluster.initial_master_nodes after the cluster is formed. Leaving it in place can cause split-brain during a full cluster restart.

Cross-Zone / Cross-AZ Deployment

Distributing nodes across availability zones (AZs) protects against zone-level failures.

ASCII Diagram — Multi-AZ Deployment

┌──────────────────────────────────────────────────────────────────┐
│                        Load Balancer                             │
│                   (Kibana + ES coordinating)                     │
└──────────┬──────────────────┬──────────────────┬─────────────────┘
           │                  │                  │
    ┌──────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
    │    AZ-1     │    │    AZ-2     │    │    AZ-3     │
    │             │    │             │    │             │
    │ es-master-1 │    │ es-master-2 │    │ es-master-3 │
    │ es-data-1a  │    │ es-data-2a  │    │ es-data-3a  │
    │ es-data-1b  │    │ es-data-2b  │    │ es-data-3b  │
    │ kibana-1    │    │ kibana-2    │    │ kibana-3    │
    │ logstash-1  │    │ logstash-2  │    │ logstash-3  │
    └─────────────┘    └─────────────┘    └─────────────┘

Shard Allocation Awareness

Use cluster.routing.allocation.awareness to spread primary and replica shards across zones:

# elasticsearch.yml — every data node
cluster.routing.allocation.awareness.attributes: zone

# On each node, set the zone attribute
node.attr.zone: az-1   # change per AZ

To force allocation awareness (prevent all copies of a shard from landing in one zone):

cluster.routing.allocation.awareness.force.zone.values: az-1,az-2,az-3

This guarantees that if only one zone is available, unassigned replicas stay unassigned rather than all landing in the surviving zone.

Replica Strategy

Replicas serve two purposes — fault tolerance and read throughput.

Scenario Recommended Replicas
2-zone deployment 1 replica (each shard has a copy in each zone)
3-zone deployment 1–2 replicas
Critical data 2 replicas (survive 2-node failures)
High read throughput 2+ replicas (more copies = more search parallelism)

Set the default on index creation:

curl -X PUT "localhost:9200/my-index" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}'

Network Partitioning Handling

Elasticsearch handles network partitions through:

  1. Fault detection — The elected master pings data nodes. If a node doesn’t respond within discovery.cluster_state_publish_timeout, it is removed.
  2. No split-brain — The quorum-based leader election prevents two masters from forming.
  3. Shard reallocation — After a node leaves, the master reallocates shards to remaining nodes (controlled by cluster.routing.allocation.node_concurrent_recoveries).

Key settings:

# elasticsearch.yml
discovery.seed_hosts:
  - es-master-1:9300
  - es-master-2:9300
  - es-master-3:9300

cluster.fault_detection.leader_check.interval: 1s
cluster.fault_detection.leader_check.timeout: 10s
cluster.fault_detection.follower_check.interval: 1s
cluster.fault_detection.follower_check.timeout: 10s

Cross-Cluster Replication (CCR)

CCR replicates indices from a leader cluster to one or more follower clusters for disaster recovery or geo-locality.

Step 1 — Configure Remote Cluster

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.remote.dc-west": {
      "seeds": ["es-west-1:9300", "es-west-2:9300"]
    }
  }
}'

Step 2 — Create Follower Index

curl -X PUT "localhost:9200/logs-follower/_ccr/follow" -H 'Content-Type: application/json' -d'
{
  "remote_cluster": "dc-west",
  "leader_index": "logs-2026.04"
}'

Step 3 — Monitor Replication

curl -s "localhost:9200/logs-follower/_ccr/stats" | jq .

Disaster Recovery Planning

Strategy RPO RTO Complexity
Replicas within cluster 0 Seconds Low
CCR to secondary cluster Near-zero Minutes Medium
Snapshots to S3/GCS Hours (depends on schedule) 30 min–hours Low
CCR + Snapshots Near-zero backup, minutes restore Minutes High

RPO = Recovery Point Objective (how much data you can afford to lose). RTO = Recovery Time Objective (how fast you need to recover).

Load Balancing

Place a load balancer in front of both Elasticsearch and Kibana:

  • Elasticsearch — Round-robin to coordinating-only nodes (or data nodes if no dedicated coordinators). Use the /_cluster/health endpoint for health checks.
  • Kibana — Round-robin across Kibana instances. All instances must point to the same ES cluster and share the same encryptionKey.

Example Nginx upstream for Elasticsearch:

upstream elasticsearch {
    least_conn;
    server es-coord-1:9200;
    server es-coord-2:9200;
    server es-coord-3:9200;
}

server {
    listen 9200;
    location / {
        proxy_pass http://elasticsearch;
        proxy_set_header Host $host;
    }
    location /_cluster/health {
        proxy_pass http://elasticsearch;
    }
}

Complete HA elasticsearch.yml

# ── Cluster ──
cluster.name: production-elk
node.name: es-data-1a

# ── Node Roles ──
node.roles: [data, ingest]

# ── Paths ──
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

# ── Network ──
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

# ── Discovery ──
discovery.seed_hosts:
  - es-master-1:9300
  - es-master-2:9300
  - es-master-3:9300

# ── Zone Awareness ──
node.attr.zone: az-1
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: az-1,az-2,az-3

# ── Recovery ──
gateway.expected_data_nodes: 6
gateway.recover_after_data_nodes: 4
cluster.routing.allocation.node_concurrent_recoveries: 4

# ── Fault Detection ──
cluster.fault_detection.leader_check.interval: 1s
cluster.fault_detection.leader_check.timeout: 10s

# ── Security (TLS) ──
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12

Quick Reference

Task Command / Setting
Check cluster health GET _cluster/health
View shard allocation GET _cat/shards?v
Check zone awareness GET _cluster/settings?include_defaults&filter_path=**.awareness*
List voting config GET _cluster/state?filter_path=metadata.cluster_coordination
CCR follower stats GET /<index>/_ccr/stats
Pause CCR POST /<index>/_ccr/pause_follow
Resume CCR POST /<index>/_ccr/resume_follow

Next Steps

With a highly available cluster in place, the next critical task is protecting your data with automated backups. Continue to Backup & Restore.

You may also enjoy