ELK Stack — FAQ

8 minute read

Frequently Asked Questions

Common questions and answers about the ELK Stack, Elasticsearch, and the broader Elastic ecosystem.

General Questions

What is the difference between ELK Stack and Elastic Stack?

ELK Stack originally referred to Elasticsearch + Logstash + Kibana. When Elastic added Beats (lightweight data shippers), they rebranded to Elastic Stack to include all four components. Today, “ELK Stack” and “Elastic Stack” are used interchangeably in the community, though “Elastic Stack” is the official name.

The Elastic Stack also includes:

  • Elastic Agent — unified agent replacing individual Beats
  • Fleet — centralized agent management
  • APM Server — application performance monitoring
  • Elastic Security — SIEM and endpoint protection

What is the licensing model for Elasticsearch?

As of Elasticsearch 7.11+, Elastic changed from Apache 2.0 to a dual license:

License Features Cost
Elastic License 2.0 Most features including security, alerting, ML Free
SSPL Server-side public license (source-available) Free
Basic (self-managed) Core features + security + ILM Free
Platinum ML, advanced security, cross-cluster replication Paid subscription
Enterprise Searchable snapshots, all features Paid subscription

Key point: Basic security (TLS, RBAC, API keys) is free since version 6.8/7.1.

What is OpenSearch and how does it differ from Elasticsearch?

OpenSearch is an AWS-maintained fork of Elasticsearch 7.10.2 (the last Apache 2.0 version):

Aspect Elasticsearch OpenSearch
Maintained by Elastic AWS (community)
License Elastic License / SSPL Apache 2.0
Latest version 8.x+ 2.x
ML features Built-in (paid) Community plugins
Security Built-in (free basic) OpenSearch Security plugin
Cloud offering Elastic Cloud Amazon OpenSearch Service
Compatibility Original APIs Fork of ES 7.10 APIs

Choose OpenSearch if you need a strictly Apache 2.0 license or are heavily invested in AWS. Choose Elasticsearch for the latest features, better performance, and official support.

How does Elasticsearch compare to Apache Solr?

Feature Elasticsearch Apache Solr
Ease of setup Simple, runs out of the box More configuration needed
API REST + JSON natively REST, XML, JSON
Scaling Auto-shard rebalancing Manual shard management
Analytics Powerful aggregations Facets and stats
Real-time Near-real-time by default Near-real-time with commit
Ecosystem Full stack (Kibana, Beats, Logstash) Limited (Banana, Solr UI)
Use case fit Log analytics, observability, search Enterprise search, e-commerce
Community Very large, active Mature, Apache community

Elasticsearch dominates for log analytics and observability. Solr remains competitive for enterprise search use cases.

Architecture Questions

What are the minimum hardware requirements?

Development / Testing:

Component CPU RAM Disk
Elasticsearch 2 cores 4 GB 20 GB SSD
Kibana 1 core 2 GB 1 GB
Logstash 2 cores 4 GB 5 GB
Filebeat 1 core 512 MB 1 GB

Production (minimum viable):

Component CPU RAM Disk
ES Master (×3) 2 cores 4 GB 10 GB SSD
ES Data (×2) 8 cores 32 GB 500 GB SSD
Kibana (×2) 4 cores 8 GB 10 GB
Logstash (×2) 4 cores 8 GB 20 GB

Rule of thumb: Elasticsearch data nodes should have at least 16 GB RAM (8 GB heap).

Should I use cloud-managed or self-managed Elasticsearch?

Factor Cloud-Managed Self-Managed
Operational overhead Low (Elastic manages) High (you manage everything)
Cost Subscription-based, predictable Infrastructure + engineering time
Customisation Limited to offering Full control
Upgrades One-click Manual rolling upgrades
Scaling Auto or one-click Manual provisioning
Data residency Cloud provider regions Your data centres
Best for Teams < 5, rapid deployment Large teams, compliance needs

Recommendation: Start with Elastic Cloud for speed, migrate to self-managed only if you have specific compliance, cost, or customisation requirements.

How many shards should I have?

Follow these guidelines:

  • Shard size: Target 10-50 GB per shard
  • Shards per node: Keep below 20 shards per GB of heap
  • Total formula: total_shards = daily_data_size × retention_days / target_shard_size

Example: 50 GB/day, 30-day retention, 25 GB target shard size:

total_shards = 50 × 30 / 25 = 60 shards
With 1 replica: 120 total shards
3 data nodes: 40 shards per node

Common mistake: Over-sharding with too many small shards. A single shard of 30 GB performs better than 30 shards of 1 GB.

Data Processing Questions

When should I use Logstash vs Beats?

Criteria Use Beats Use Logstash
Parsing needed No or minimal Complex (Grok, multi-step)
Transformation Simple processors Heavy transformation
Resource constraints Limited (edge) Dedicated processing server
Multiple outputs Single output Multiple outputs, routing
Enrichment Basic metadata Lookups, JDBC, GeoIP
Buffering Limited Persistent queues
Scale One per host Centralised, scale horizontally

Common pattern: Use Beats for collection → Logstash for processing → Elasticsearch for storage.

Modern alternative: Use Beats + Elasticsearch Ingest Pipelines (no Logstash needed for many use cases).

How do I handle high-cardinality fields?

High-cardinality fields (e.g., user IDs, IP addresses, session tokens) can cause memory issues:

Problems:

  • terms aggregations on high-cardinality fields use significant memory
  • fielddata on text fields can cause circuit breaker exceptions

Solutions:

// 1. Use keyword type, not text, for aggregations
PUT /my-index
{
  "mappings": {
    "properties": {
      "user_id": { "type": "keyword" }
    }
  }
}

// 2. Use composite aggregation for paginated results
GET /my-index/_search
{
  "size": 0,
  "aggs": {
    "unique_users": {
      "composite": {
        "size": 1000,
        "sources": [
          { "user": { "terms": { "field": "user_id" } } }
        ]
      }
    }
  }
}

// 3. Use cardinality aggregation for approximate counts
GET /my-index/_search
{
  "size": 0,
  "aggs": {
    "unique_count": {
      "cardinality": {
        "field": "user_id",
        "precision_threshold": 10000
      }
    }
  }
}

How do I reduce storage costs?

  1. Enable compression: 
    PUT /my-index/_settings
{
  "index.codec": "best_compression"
}

  2. Use ILM to move data to cheaper storage tiers (hot → warm → cold → frozen)

  3. Reduce replicas for older indices: 
    PUT /my-index/_settings
{
  "number_of_replicas": 0
}
  4. Force merge read-only indices: 
    curl -X POST "localhost:9200/my-old-index/_forcemerge?max_num_segments=1"

  5. Use searchable snapshots for cold/frozen tier (store in S3/GCS at ~80% cost reduction)

  6. Disable _source if you don’t need the original document (caution: you lose reindex ability)

  7. Optimise mappings — disable indexing for fields you don’t search, use appropriate types

Troubleshooting Questions

My cluster status is yellow — what should I do?

Yellow means all primary shards are allocated, but some replicas are not.

Common causes:

  1. Single-node cluster — replicas can’t be allocated on the same node: 
    PUT /my-index/_settings
{
  "number_of_replicas": 0
}

  2. Insufficient nodes — add more data nodes or reduce replicas

  3. Disk watermark — check disk usage: 
    curl -s "localhost:9200/_cat/allocation?v"
  4. Shard allocation filtering — check if allocation rules prevent placement: 
    curl -s "localhost:9200/_cluster/allocation/explain?pretty"

My cluster status is red — what should I do?

Red means one or more primary shards are unassigned. This is serious.

Immediate steps:

# 1. Check which indices are red
curl -s "localhost:9200/_cat/indices?health=red&v"

# 2. Find unassigned shards
curl -s "localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason" | grep UNASSIGNED

# 3. Get allocation explanation
curl -s "localhost:9200/_cluster/allocation/explain?pretty"

# 4. Check node status
curl -s "localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,disk.used_percent,node.role"

Common fixes:

  • Restart failed nodes
  • Reroute stuck shards: POST /_cluster/reroute?retry_failed=true
  • Allocate empty primary (DATA LOSS): only as last resort

Elasticsearch is using too much heap memory — how do I fix it?

Diagnosis:

# Check heap usage
curl -s "localhost:9200/_nodes/stats/jvm?pretty" | grep -A 5 "heap"

# Check circuit breakers
curl -s "localhost:9200/_nodes/stats/breaker?pretty"

Solutions:

  1. Increase heap (up to 50% of RAM, max ~31 GB): 
    # jvm.options
-Xms16g
-Xmx16g

  2. Reduce fielddata — avoid text field aggregations, use keyword type

  3. Optimise queries — use filter context for caching, avoid deep pagination

  4. Reduce shard count — fewer shards means less overhead

  5. Use doc_values (default) instead of fielddata for sorting/aggregations

Search queries are slow — how do I diagnose?

Step 1: Enable slow log:

# elasticsearch.yml (or via API)
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.fetch.warn: 1s

Step 2: Use the Profile API:

GET /my-index/_search
{
  "profile": true,
  "query": {
    "match": { "message": "error" }
  }
}

Step 3: Common optimisations:

  • Use filter context instead of query for exact matches
  • Add routing to target specific shards
  • Use _source filtering to return only needed fields
  • Avoid wildcard queries with leading wildcards
  • Pre-sort indices for common sort patterns
  • Consider terminate_after for exploratory queries

How do I migrate from ELK to Elastic Cloud?

  1. Create Elastic Cloud deployment matching your current version
  2. Set up snapshot repository accessible by both clusters (S3/GCS)
  3. Snapshot your data: 
    curl -X PUT "localhost:9200/_snapshot/migration/snapshot_1?wait_for_completion=true"
  4. Register the same repository on Elastic Cloud
  5. Restore the snapshot on Elastic Cloud
  6. Update Beats/Logstash to point to the new Cloud cluster
  7. Verify data and switch DNS/load balancer
  8. Decommission the old cluster

How do I handle timezone issues in logs?

Problem: Logs arrive with different timezones, causing confusion in Kibana.

Solutions:

  1. Standardise on UTC at the source when possible

  2. Parse and convert in Logstash: 
    filter {
  date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
    timezone => "America/New_York"
    target => "@timestamp"
  }
}

  3. Use Kibana timezone setting: Management → Advanced Settings → dateFormat:tz

  4. Store original timezone in a separate field for reference

Operational Questions

How often should I upgrade Elasticsearch?

  • Minor releases (8.12 → 8.13): Every 1-3 months, mainly for bug fixes and features
  • Major releases (7.x → 8.x): Plan carefully, test in staging first
  • Security patches: Apply immediately
  • Always read release notes and run the Upgrade Assistant in Kibana before upgrading

Can I run Elasticsearch on Docker in production?

Yes, with caveats:

  • Set vm.max_map_count=262144 on the Docker host
  • Use volume mounts for data (not Docker volumes for performance)
  • Set ES_JAVA_OPTS explicitly
  • Use ECK (Elastic Cloud on Kubernetes) for container orchestration
  • Ensure proper resource limits (memory, CPU)
  • Pin specific Elasticsearch image versions (don’t use latest)

How do I handle index rollover for time-series data?

Use Data Streams (recommended in 8.x):

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  }
}

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Data streams automatically handle rollover, backing indices, and the write index.

Have a question not covered here? Check the Community Resources page for forums and support channels.

You may also enjoy