ELK Stack — Cluster Architecture

Cluster Architecture

A well-designed cluster separates concerns across node roles, sizes hardware to match workloads, and plans for growth. This section covers every node role, explains when to use each, and shows how to build a production-grade topology.

---

Node Roles

Elasticsearch 8.x uses the node.roles setting to assign one or more roles to each node.

Role	Purpose
master	Manages cluster state, index creation, shard allocation
data	Stores data and executes searches and aggregations
data_hot	Stores frequently queried, recent data on fast storage
data_warm	Stores less-frequently queried data on standard storage
data_cold	Stores rarely queried data, may use searchable snapshots
data_frozen	Stores data entirely in snapshots, fetched on demand
ingest	Runs ingest pipelines before indexing
coordinating	Routes requests, merges search results (no data, no master)
ml	Runs machine learning jobs
transform	Runs transform jobs (pivot and latest)
remote_cluster_client	Connects to remote clusters for cross-cluster search

Assigning Roles

# elasticsearch.yml — dedicated master node
node.roles: [ master ]

# elasticsearch.yml — hot data node
node.roles: [ data_hot, ingest ]

# elasticsearch.yml — warm data node
node.roles: [ data_warm ]

# elasticsearch.yml — cold data node
node.roles: [ data_cold ]

# elasticsearch.yml — coordinating-only node
node.roles: [ ]

# elasticsearch.yml — ML node
node.roles: [ ml, remote_cluster_client ]

Note: A node with node.roles: [ ] (empty array) acts as a coordinating-only node. It holds no data and is not master-eligible, but it can route and merge requests.

---

Dedicated Master Nodes

Master nodes manage the cluster state — the metadata describing every index, shard, mapping, and setting. In production you must run at least 3 dedicated master-eligible nodes to avoid split-brain and guarantee high availability.

Why 3?

Elasticsearch uses a quorum-based consensus algorithm. With 3 master-eligible nodes, the cluster tolerates the loss of 1 node and still elects a master (quorum = 2).

Master-eligible nodes	Quorum	Tolerated failures
1	1	0
3	2	1
5	3	2

Master Node Sizing

Dedicated masters are lightweight because they never hold data:

• CPU: 2–4 cores
• RAM: 4–8 GB (1–2 GB heap is often sufficient)
• Disk: 10 GB SSD (cluster state only)
• Network: Low latency to all other nodes

# elasticsearch.yml — dedicated master
node.roles: [ master ]
node.name: master-01
cluster.name: prod-cluster

# Initial master nodes for bootstrapping
cluster.initial_master_nodes:
  - master-01
  - master-02
  - master-03

Warning: Never run data workloads on dedicated master nodes. A GC pause or heavy I/O on a master can delay shard allocation across the entire cluster.

---

Data Node Sizing

Data nodes do the heavy lifting — indexing, searching, and aggregating. Size them based on your workload profile:

Storage-Heavy (Logging / Time-Series)

Resource	Recommendation
CPU	8–16 cores
RAM	32–64 GB
Heap	50% of RAM, max 31 GB
Disk	2–8 TB NVMe/SSD per node
Shards	~20 shards per GB of heap

Search-Heavy (E-Commerce / App Search)

Resource	Recommendation
CPU	16–32 cores
RAM	64 GB
Heap	31 GB
Disk	1–4 TB NVMe
Shards	Fewer, larger shards for better cache locality

---

Coordinating-Only Nodes

Coordinating nodes act as smart load balancers within the cluster. They:

1. Receive client requests
2. Route requests to the correct data nodes
3. Merge partial results (scatter-gather)
4. Return the final response to the client

Use coordinating nodes when:

• You have heavy aggregation queries that require significant merge work
• You want to isolate client-facing traffic from data nodes
• Your cluster has 10+ data nodes

# elasticsearch.yml — coordinating-only node
node.roles: [ ]
node.name: coord-01

---

Split-Brain Prevention

Elasticsearch 7.x+ uses an automatic quorum-based voting system that eliminates the old minimum_master_nodes setting. However, you should still:

1. Always use an odd number of master-eligible nodes (3 or 5)
2. Never run master-eligible roles on data nodes in production
3. Use cluster.initial_master_nodes only during first cluster bootstrap — remove it afterward
4. Ensure low-latency networking between master-eligible nodes (< 10 ms RTT)

---

Cluster State

The cluster state contains:

• Node membership and roles
• Index metadata (settings, mappings, aliases)
• Shard allocation table
• Ingest pipeline definitions
• ILM policies and templates

Monitor cluster state size:

# Cluster state size
curl -s localhost:9200/_cluster/state?filter_path=metadata.indices.*.settings | wc -c

# Cluster health
curl -s localhost:9200/_cluster/health?pretty

{
  "cluster_name": "prod-cluster",
  "status": "green",
  "number_of_nodes": 12,
  "number_of_data_nodes": 6,
  "active_primary_shards": 450,
  "active_shards": 900,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0
}

Tip: If your cluster state exceeds 100 MB, you likely have too many small indices or too many fields. Consider using data streams and reducing mapping complexity.

---

Hot-Warm-Cold Architecture

This tiered architecture moves data through storage tiers based on age and query frequency, balancing performance with cost.

┌─────────────────────────────────────────────────────────────┐
│                    PRODUCTION CLUSTER                        │
│                                                             │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐                     │
│  │master-01│  │master-02│  │master-03│   Dedicated Masters  │
│  └─────────┘  └─────────┘  └─────────┘                     │
│                                                             │
│  ┌──────────────────────────────────────────┐               │
│  │          HOT TIER (NVMe SSD)             │               │
│  │  ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐   │               │
│  │  │hot-01│ │hot-02│ │hot-03│ │hot-04│   │  Recent data   │
│  │  └──────┘ └──────┘ └──────┘ └──────┘   │  (0-7 days)   │
│  └──────────────────────────────────────────┘               │
│                      ▼ ILM rollover                         │
│  ┌──────────────────────────────────────────┐               │
│  │         WARM TIER (SSD / HDD)            │               │
│  │  ┌───────┐ ┌───────┐ ┌───────┐         │               │
│  │  │warm-01│ │warm-02│ │warm-03│         │  Older data    │
│  │  └───────┘ └───────┘ └───────┘         │  (7-30 days)  │
│  └──────────────────────────────────────────┘               │
│                      ▼ ILM migrate                          │
│  ┌──────────────────────────────────────────┐               │
│  │         COLD TIER (HDD / Object)         │               │
│  │  ┌───────┐ ┌───────┐                    │               │
│  │  │cold-01│ │cold-02│                    │  Archive data  │
│  │  └───────┘ └───────┘                    │  (30-90 days) │
│  └──────────────────────────────────────────┘               │
│                      ▼ ILM searchable snapshot              │
│  ┌──────────────────────────────────────────┐               │
│  │       FROZEN TIER (Snapshot repo)        │               │
│  │  ┌─────────┐ ┌─────────┐               │               │
│  │  │frozen-01│ │frozen-02│               │  90+ days      │
│  │  └─────────┘ └─────────┘               │  (on demand)  │
│  └──────────────────────────────────────────┘               │
│                                                             │
│  ┌──────────┐ ┌──────────┐                                  │
│  │ coord-01 │ │ coord-02 │   Coordinating / Load Balancer   │
│  └──────────┘ └──────────┘                                  │
└─────────────────────────────────────────────────────────────┘

Configuring Tiers

# hot node
node.roles: [ data_hot, ingest ]
node.attr.box_type: hot

# warm node
node.roles: [ data_warm ]
node.attr.box_type: warm

# cold node
node.roles: [ data_cold ]
node.attr.box_type: cold

Use ILM to move indices between tiers automatically:

curl -X PUT "localhost:9200/_ilm/policy/logs-lifecycle" \
  -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "my-repo" },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}'

---

Quick Reference

Decision	Recommendation
Master nodes	3 dedicated, never on data nodes
Data tiers	Hot → Warm → Cold → Frozen via ILM
Coordinating nodes	Add when 10+ data nodes or heavy aggs
Split-brain	Odd number of masters, low-latency network
Cluster state	Keep under 100 MB; reduce indices/fields
Node heap	50% of RAM, max 31 GB
Bootstrap	Remove cluster.initial_master_nodes after first start

---

← Performance and Scaling · Next → Shard Strategy