ELK Stack — What is ELK Stack?
What is the ELK Stack?
ELK stands for Elasticsearch, Logstash, and Kibana — three open-source tools that work together to collect, store, search, and visualize log data. Think of it as a Google for your logs — except you own the search engine and control every piece of data.
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ │ │ │ │ │
│ Logstash │────▶│Elasticsearch│────▶│ Kibana │
│ (Collect & │ │ (Store & │ │ (Visualize │
│ Transform) │ │ Search) │ │ & Explore) │
│ │ │ │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
Here’s what each letter stands for:
- E — Elasticsearch: A distributed search and analytics engine. It stores your logs and lets you search through millions of records in milliseconds.
- L — Logstash: A data processing pipeline. It collects logs from various sources, transforms them (parsing, filtering, enriching), and sends them to Elasticsearch.
- K — Kibana: A visualization platform. It connects to Elasticsearch and lets you build dashboards, charts, and explore your data interactively.
The Origin Story
The ELK Stack wasn’t planned as a unified product. Each tool was created independently:
| Year | Event |
|---|---|
| 2004 | Shay Banon created Compass, a Java search framework |
| 2010 | Banon rewrote Compass as Elasticsearch — a distributed, RESTful search engine built on Apache Lucene |
| 2011 | Jordan Sissel created Logstash — a tool to collect and process logs |
| 2013 | Rashid Khan created Kibana — a visualization frontend for Elasticsearch |
| 2014 | Elastic (the company) brought all three tools together as the ELK Stack |
| 2015 | Beats (lightweight data shippers) were introduced, and the stack became the Elastic Stack |
| 2021 | Elasticsearch license changed from Apache 2.0 to SSPL/Elastic License |
| 2024 | Elasticsearch returned to open source under AGPL |
Why Does ELK Exist?
Before the ELK Stack, engineers faced a common problem:
The Log Chaos Problem:
┌──────────────────────────────────────────────────────┐
│ Before ELK │
│ │
│ Web Server Logs ──▶ /var/log/nginx/access.log │
│ App Server Logs ──▶ /var/log/app/application.log │
│ Database Logs ──▶ /var/log/mysql/error.log │
│ Auth Logs ──▶ /var/log/auth.log │
│ Container Logs ──▶ docker logs <container> │
│ │
│ ❌ Logs scattered across 50+ servers │
│ ❌ SSH into each server to read logs │
│ ❌ grep through gigabytes of text │
│ ❌ No correlation between services │
│ ❌ No way to search across all logs at once │
└──────────────────────────────────────────────────────┘
The ELK Solution:
┌──────────────────────────────────────────────────────┐
│ With ELK │
│ │
│ Web Server Logs ──┐ │
│ App Server Logs ──┤ │
│ Database Logs ──┼──▶ Logstash ──▶ Elasticsearch │
│ Auth Logs ──┤ ▲ │
│ Container Logs ──┘ │ │
│ Kibana │
│ ✅ All logs in one place │
│ ✅ Search across everything instantly │
│ ✅ Correlate events across services │
│ ✅ Beautiful dashboards and alerts │
│ ✅ Retain and analyze historical data │
└──────────────────────────────────────────────────────┘
ELK vs Elastic Stack — What’s the Difference?
You’ll hear both terms used interchangeably, but there’s a subtle difference:
| Term | Includes |
|---|---|
| ELK Stack | Elasticsearch + Logstash + Kibana (the original trio) |
| Elastic Stack | Elasticsearch + Logstash + Kibana + Beats + Elastic Agent (the modern suite) |
The term ELK Stack stuck as a brand name even though the modern stack includes Beats and Elastic Agent. Throughout this tutorial, we’ll use “ELK Stack” to refer to the complete Elastic Stack ecosystem.
How Big Is ELK?
The ELK Stack is used by thousands of organizations worldwide:
- Netflix — Monitors streaming infrastructure with billions of log events per day
- LinkedIn — Processes petabytes of log data for search and analytics
- Uber — Uses Elasticsearch for real-time trip monitoring and fraud detection
- Wikipedia — Powers the site search using Elasticsearch
- GitHub — Searches through billions of lines of code
- Stack Overflow — Uses Elasticsearch for question search
It’s not just for big tech — any organization with servers, applications, or infrastructure can benefit from centralized logging with ELK.
Key Terminology
Before we dive deeper, here are the essential terms you’ll encounter throughout this tutorial:
| Term | Definition |
|---|---|
| Index | A collection of documents in Elasticsearch (like a database table) |
| Document | A single JSON record stored in an index (like a row in a table) |
| Shard | A subset of an index — Elasticsearch splits large indices across shards for scalability |
| Pipeline | A Logstash configuration that defines how data is collected, transformed, and output |
| Beat | A lightweight agent that ships specific types of data (logs, metrics, uptime) |
| Dashboard | A Kibana page with multiple visualizations arranged together |
| Query DSL | Elasticsearch’s JSON-based query language for searching data |
| Mapping | The schema definition for an Elasticsearch index |
Don’t worry if these terms feel overwhelming right now. Each one will be explained in detail with hands-on examples in the upcoming chapters.
What Version Are We Using?
This tutorial is based on Elasticsearch 8.x, Kibana 8.x, Logstash 8.x, and Filebeat 8.x. Version 8 introduced significant changes including:
- Security enabled by default (TLS + authentication)
- Simplified REST API
- New Kibana UI with improved Discover and Lens
- Better support for Elastic Agent and Fleet
All commands and configurations in this tutorial are tested against Elasticsearch 8.x.
Quick Reference
Concept Key Point ELK stands for Elasticsearch + Logstash + Kibana Created by Elastic (originally Shay Banon) Purpose Centralized log collection, search, and visualization Modern name Elastic Stack (includes Beats + Elastic Agent) Version used 8.x (security enabled by default) License AGPL (open source as of 2024)
Next Step
Now that you know what the ELK Stack is, let’s look at how the architecture works — how data actually flows from your servers to a Kibana dashboard.
