ELK Stack — Upgrade Strategies

5 minute read

Upgrade Strategies for the ELK Stack

Keeping Elasticsearch, Kibana, Logstash, and Beats up to date is critical for security patches, bug fixes, and new features. This section covers rolling upgrades, full-cluster restarts, blue-green upgrades, and the complete pre-upgrade checklist.

Upgrade Path & Version Compatibility

Elasticsearch supports rolling upgrades between consecutive minor versions within the same major version (e.g., 8.10 → 8.11 → 8.12). Major version upgrades (e.g., 7.17 → 8.x) require specific paths:

From To Method
7.17 → 8.x Latest 8.x Rolling upgrade (if from 7.17)
7.x (< 7.17) → 8.x 7.17 first, then 8.x Two-step rolling upgrade
6.8 → 7.x 7.17 Rolling upgrade
6.x (< 6.8) → 7.x 6.8 first Two-step
5.x → 7.x 5.6 → 6.8 → 7.17 Three-step

Rule: Always upgrade to the latest minor version of your current major before crossing to the next major.

Component Upgrade Order

Upgrade components in this order:

  1. Elasticsearch (data nodes last, masters first)
  2. Kibana
  3. Logstash
  4. Beats (Filebeat, Metricbeat, etc.)
  5. APM Server

Important: Kibana must match the Elasticsearch version exactly. Beats and Logstash can be one major version behind ES during the upgrade window.

Pre-Upgrade Checklist

Complete every item before starting:

1. Check Deprecation API

curl -s "localhost:9200/_migration/deprecations" | jq .

This returns deprecated settings, index settings, and API usages that must be resolved before upgrading.

2. Run Upgrade Assistant in Kibana

Navigate to Stack Management → Upgrade Assistant. It highlights:

  • Deprecated index settings
  • Incompatible mappings
  • Deprecated cluster settings
  • Reindex requirements

3. Take a Full Snapshot

curl -X PUT "localhost:9200/_snapshot/my_repo/pre-upgrade-$(date +%Y%m%d)?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "indices": "*",
  "include_global_state": true
}'

4. Verify Snapshot

curl -s "localhost:9200/_snapshot/my_repo/pre-upgrade-*" | jq '.snapshots[0].state'
# Should return "SUCCESS"

5. Check Cluster Health

curl -s "localhost:9200/_cluster/health" | jq '.status'
# Must be "green"

6. Review Release Notes

Read the Elasticsearch release notes for breaking changes.

7. Test in Staging

Always test the upgrade on a staging cluster with production-like data before upgrading production.

Rolling Upgrade — Step by Step

A rolling upgrade upgrades one node at a time with zero downtime.

Step 1 — Disable Shard Allocation

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

Step 2 — Stop Non-Essential Indexing (Optional)

Pause Logstash pipelines or reduce Beats output if possible to speed up recovery.

Step 3 — Perform a Synced Flush (ES < 8.0)

curl -X POST "localhost:9200/_flush/synced"

Note: Synced flush is deprecated in 7.6+ and removed in 8.0. In 8.x, a normal flush is sufficient.

Step 4 — Stop and Upgrade One Node

# On the target node
sudo systemctl stop elasticsearch

# Upgrade the package
# Debian/Ubuntu
sudo apt-get update && sudo apt-get install elasticsearch=8.12.0

# RHEL/CentOS
sudo yum install elasticsearch-8.12.0

# Start the node
sudo systemctl start elasticsearch

Step 5 — Wait for the Node to Join

curl -s "localhost:9200/_cat/nodes?v&h=name,version"

Wait until the upgraded node appears in the node list.

Step 6 — Re-Enable Shard Allocation

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}'

Step 7 — Wait for Green

curl -s "localhost:9200/_cluster/health?wait_for_status=green&timeout=120s"

Step 8 — Repeat for Each Node

Repeat Steps 1–7 for every node. Upgrade master-eligible nodes first, then data nodes.

Upgrading Kibana

  1. Stop Kibana:
sudo systemctl stop kibana
  1. Upgrade the package:
# Debian/Ubuntu
sudo apt-get install kibana=8.12.0

# RHEL/CentOS
sudo yum install kibana-8.12.0
  1. Start Kibana:
sudo systemctl start kibana
  1. Verify:
curl -s "localhost:5601/api/status" | jq '.version.number'

Note: Kibana performs saved-object migrations on startup after a major version upgrade. This can take several minutes for large deployments.

Upgrading Logstash

sudo systemctl stop logstash
sudo apt-get install logstash=1:8.12.0-1   # Debian
sudo systemctl start logstash

Check pipeline health:

curl -s "localhost:9600/_node/stats" | jq '.pipelines'

Upgrading Beats

sudo systemctl stop filebeat
sudo apt-get install filebeat=8.12.0
sudo systemctl start filebeat

Repeat for each Beat (Metricbeat, Heartbeat, Packetbeat, etc.).

Full Cluster Restart Upgrade

Required when a rolling upgrade is not supported (e.g., skipping minor versions or specific major transitions).

When Is It Needed?

  • Upgrading from ES 6.x to 7.x (if not on 6.8)
  • Custom plugins that are incompatible during mixed-version operation

Steps

  1. Complete the pre-upgrade checklist (snapshot, deprecation check).
  2. Disable shard allocation and flush:
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'
curl -X POST "localhost:9200/_flush"
  1. Stop all nodes.
  2. Upgrade all nodes.
  3. Start master-eligible nodes first, wait for quorum.
  4. Start data nodes.
  5. Re-enable shard allocation:
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}'
  1. Wait for green.

Blue-Green Upgrade Approach

For zero-risk upgrades, build a new cluster alongside the existing one:

┌──────────────┐         ┌──────────────┐
│  Blue (Old)  │  ─CCR→  │  Green (New) │
│   ES 8.10    │         │   ES 8.12    │
│   Kibana     │         │   Kibana     │
└──────────────┘         └──────────────┘
        ▲                        ▲
        │                        │
   Current traffic          Test traffic

Steps

  1. Provision the Green cluster with the target version.
  2. Set up Cross-Cluster Replication (CCR) from Blue → Green.
  3. Wait for all follower indices to catch up.
  4. Test the Green cluster thoroughly.
  5. Switch traffic (update load balancer or DNS) to Green.
  6. Promote follower indices to regular indices on Green.
  7. Decommission the Blue cluster.

Advantages

  • Zero downtime, zero risk — roll back by switching traffic back.
  • Clean environment — no mixed-version nodes.

Disadvantages

  • Doubles infrastructure cost during the migration window.
  • Requires CCR license (Platinum/Enterprise or Elastic Cloud).

Post-Upgrade Verification

After upgrading all components:

# 1. Check versions
curl -s "localhost:9200/_cat/nodes?v&h=name,version"
curl -s "localhost:5601/api/status" | jq '.version.number'

# 2. Cluster health
curl -s "localhost:9200/_cluster/health" | jq .

# 3. Run a test search
curl -s "localhost:9200/logs-*/_search?size=1" | jq '.hits.total'

# 4. Check for deprecation warnings in logs
grep -i "deprecat" /var/log/elasticsearch/production-elk.log | tail -20

# 5. Validate dashboards in Kibana
# Open key dashboards and verify data is flowing

Quick Reference

Task Command / Action
Check deprecations GET _migration/deprecations
Disable shard allocation PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "primaries" } }
Re-enable allocation PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": null } }
Synced flush (< 8.0) POST _flush/synced
Flush (8.0+) POST _flush
Check node versions GET _cat/nodes?v&h=name,version
Snapshot before upgrade PUT _snapshot/repo/pre-upgrade-YYYYMMDD
Kibana upgrade assistant Stack Management → Upgrade Assistant

Next Steps

Test your production knowledge with hands-on exercises. Continue to Practice Exercises.

You may also enjoy