ELK Stack — ELK Architecture
ELK Architecture Overview
Understanding how data flows through the ELK Stack is essential before you start building anything. The architecture follows a simple pipeline: Collect → Transform → Store → Visualize.
┌──────────────────────────────────────────────────────────────────┐
│ ELK Stack Data Flow │
│ │
│ ┌─────────┐ ┌─────────┐ ┌──────────────┐ ┌─────────┐ │
│ │ Beats │───▶│Logstash │───▶│Elasticsearch │───▶│ Kibana │ │
│ │(Collect)│ │(Process)│ │ (Store) │ │(Display)│ │
│ └─────────┘ └─────────┘ └──────────────┘ └─────────┘ │
│ │
│ Filebeat Grok Parse Index Data Dashboards │
│ Metricbeat Filter Full-Text Search Discover │
│ Heartbeat Enrich Aggregate Alerting │
│ Packetbeat Route Retain Lens │
└──────────────────────────────────────────────────────────────────┘
Let’s break down each stage.
Stage 1 — Data Collection (Beats)
Beats are lightweight agents installed on your servers. Their only job is to read log files (or collect metrics) and ship them to the next stage. Think of Beats as postal workers — they pick up the mail and deliver it.
┌──────────────────────────────────────────┐
│ Your Servers │
│ │
│ Server 1: [Filebeat] → reads access.log │
│ Server 2: [Filebeat] → reads app.log │
│ Server 3: [Metricbeat] → reads CPU/RAM │
│ Server 4: [Heartbeat] → pings services │
│ │
│ All Beats ship data to ──▶ Logstash │
│ or ──▶ Elasticsearch│
└──────────────────────────────────────────┘
Key characteristics of Beats:
- Lightweight — minimal CPU and memory footprint
- Purpose-built — each Beat handles one type of data
- Reliable — built-in backpressure handling and delivery guarantees
- Simple — YAML configuration, no coding required
The most common Beat is Filebeat, which tails log files and sends each line as an event.
Stage 2 — Data Processing (Logstash)
Logstash receives raw log data and transforms it into structured, searchable documents. Think of Logstash as a factory — raw materials go in, finished products come out.
A Logstash pipeline has three sections:
┌───────────────────────────────────────────────┐
│ Logstash Pipeline │
│ │
│ ┌─────────┐ ┌──────────┐ ┌────────────┐ │
│ │ INPUT │──▶│ FILTER │──▶│ OUTPUT │ │
│ │ │ │ │ │ │ │
│ │ beats │ │ grok │ │ elasticsearch│
│ │ file │ │ date │ │ stdout │ │
│ │ syslog │ │ mutate │ │ file │ │
│ │ kafka │ │ geoip │ │ kafka │ │
│ └─────────┘ └──────────┘ └────────────┘ │
└───────────────────────────────────────────────┘
Example transformation:
Raw NGINX log line:
192.168.1.50 - - [15/Mar/2025:10:15:30 +0000] "GET /api/users HTTP/1.1" 200 1234
After Logstash grok parsing:
{
"client_ip": "192.168.1.50",
"timestamp": "2025-03-15T10:15:30.000Z",
"method": "GET",
"path": "/api/users",
"http_version": "1.1",
"status_code": 200,
"bytes_sent": 1234
}
Now you can search for all status_code: 500 errors or aggregate traffic by client_ip. That’s the power of structured data.
Stage 3 — Data Storage (Elasticsearch)
Elasticsearch is the heart of the stack. It stores every log event as a JSON document and makes it searchable in near real-time (typically within 1 second of ingestion).
┌──────────────────────────────────────────────┐
│ Elasticsearch Cluster │
│ │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Node 1 │ │ Node 2 │ │
│ │ (Master) │ │ (Data) │ │
│ │ │ │ │ │
│ │ Shard 0 (P) │ │ Shard 0 (R) │ │
│ │ Shard 1 (R) │ │ Shard 1 (P) │ │
│ └─────────────┘ └─────────────┘ │
│ │
│ Index: nginx-logs-2025.03.15 │
│ Documents: 2,500,000 │
│ Size: 1.2 GB │
│ Searchable in: < 1 second │
└──────────────────────────────────────────────┘
Key capabilities:
- Full-text search — find log entries by any keyword
- Structured queries — filter by status code, IP, timestamp, or any field
- Aggregations — count errors per hour, average response time, top URLs
- Near real-time — data is searchable within ~1 second of indexing
- Distributed — scales horizontally by adding more nodes
- Resilient — data is replicated across nodes for fault tolerance
Stage 4 — Visualization (Kibana)
Kibana is the window into your data. It connects to Elasticsearch and provides a web interface to search, analyze, and visualize your logs.
┌──────────────────────────────────────────────┐
│ Kibana Features │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Discover │ │ Lens │ │Dashboards│ │
│ │ │ │ │ │ │ │
│ │ Search & │ │ Drag-drop│ │ Combined │ │
│ │ explore │ │ charts │ │ views │ │
│ │ raw logs │ │ │ │ │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Alerting │ │ Canvas │ │ Maps │ │
│ │ │ │ │ │ │ │
│ │ Notify │ │ Custom │ │ GeoIP │ │
│ │ on │ │ pixel- │ │ based │ │
│ │ anomalies│ │ perfect │ │ visuals │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└──────────────────────────────────────────────┘
Common Architecture Patterns
Pattern 1 — Simple (Small Teams)
For small setups (< 10 servers), you can skip Logstash entirely:
Filebeat ──────────▶ Elasticsearch ──▶ Kibana
Filebeat can parse common log formats using modules (NGINX, Apache, MySQL, etc.) and send structured data directly to Elasticsearch.
Pattern 2 — Standard (Most Teams)
For medium setups that need custom parsing:
Filebeat ──▶ Logstash ──▶ Elasticsearch ──▶ Kibana
Logstash handles complex parsing, enrichment, and routing.
Pattern 3 — Buffered (High Volume)
For high-volume environments (millions of events per second), add a message queue:
Filebeat ──▶ Kafka ──▶ Logstash ──▶ Elasticsearch ──▶ Kibana
Kafka acts as a buffer — if Elasticsearch is slow or down, events queue in Kafka instead of being lost.
Pattern 4 — Multi-Cluster (Enterprise)
For large organizations:
┌─────────────────────────────────────────────┐
│ Region: US-East │
│ Filebeat ──▶ Logstash ──▶ ES Cluster 1 │
└──────────────────────────┬──────────────────┘
│
Cross-Cluster Search
│
┌──────────────────────────┴──────────────────┐
│ Region: EU-West │
│ Filebeat ──▶ Logstash ──▶ ES Cluster 2 │
└─────────────────────────────────────────────┘
│
┌─────┴─────┐
│ Kibana │
│ (Central) │
└───────────┘
Data Flow — Step by Step
Let’s trace a single log event through the entire stack:
1. An NGINX server handles a request and writes to access.log:
192.168.1.50 - - [15/Mar/2025:10:15:30 +0000] "GET /api/users HTTP/1.1" 200 1234
2. Filebeat (running on the NGINX server) reads the new line and sends it to Logstash:
{
"message": "192.168.1.50 - - [15/Mar/2025:10:15:30 +0000] \"GET /api/users HTTP/1.1\" 200 1234",
"host": "web-server-01",
"log": { "file": { "path": "/var/log/nginx/access.log" } }
}
3. Logstash applies a grok filter to parse the raw message:
{
"client_ip": "192.168.1.50",
"timestamp": "2025-03-15T10:15:30.000Z",
"method": "GET",
"path": "/api/users",
"status_code": 200,
"bytes_sent": 1234,
"host": "web-server-01"
}
4. Logstash sends the structured document to Elasticsearch:
- Indexed into
nginx-logs-2025.03.15 - Searchable within ~1 second
5. You open Kibana and search for status_code: 500 to find errors:
- Kibana sends the query to Elasticsearch
- Elasticsearch returns matching documents
- Kibana displays results in the Discover view or a dashboard
Port Reference
| Component | Default Port | Purpose |
|---|---|---|
| Elasticsearch | 9200 | REST API (HTTP) |
| Elasticsearch | 9300 | Node-to-node communication |
| Kibana | 5601 | Web UI |
| Logstash (Beats input) | 5044 | Receives data from Beats |
| Logstash (HTTP input) | 9600 | Monitoring API |
| Filebeat | — | No listening port (pushes data) |
Quick Reference
Stage Component Role Collect Beats (Filebeat, Metricbeat) Ship logs/metrics from servers Transform Logstash Parse, filter, enrich log data Store Elasticsearch Index and search documents Visualize Kibana Dashboards, search, alerts Default ports ES: 9200, Kibana: 5601, Logstash: 5044 Patterns Simple, Standard, Buffered, Multi-Cluster Choose based on scale
Next Step
Now that you understand how the ELK architecture works, let’s explore real-world use cases where the ELK Stack is commonly deployed.
