ELK Stack — Elastic Stack Components

5 minute read

Elastic Stack Components

The modern Elastic Stack includes more than just Elasticsearch, Logstash, and Kibana. Let’s explore every component you’ll encounter.

The Core Components

┌─────────────────────────────────────────────────────────┐
│                  Elastic Stack Components                 │
│                                                          │
│  ┌──────────────────────────────────────────────────┐   │
│  │              Data Collection Layer                │   │
│  │  Filebeat │ Metricbeat │ Heartbeat │ Elastic Agent│   │
│  └────────────────────────┬─────────────────────────┘   │
│                           │                              │
│  ┌────────────────────────▼─────────────────────────┐   │
│  │              Data Processing Layer                │   │
│  │  Logstash │ Ingest Pipelines │ Elastic Agent      │   │
│  └────────────────────────┬─────────────────────────┘   │
│                           │                              │
│  ┌────────────────────────▼─────────────────────────┐   │
│  │              Storage & Search Layer               │   │
│  │              Elasticsearch Cluster                │   │
│  └────────────────────────┬─────────────────────────┘   │
│                           │                              │
│  ┌────────────────────────▼─────────────────────────┐   │
│  │              Visualization Layer                  │   │
│  │  Kibana │ Elastic Maps │ Canvas │ Lens            │   │
│  └──────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Elasticsearch

Role: Store, search, and analyze data.

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. It’s the core of the Elastic Stack — everything else either feeds data into it or reads data from it.

Key features:

  • Full-text search — search through unstructured text with relevance scoring
  • Structured queries — filter by exact values, ranges, and boolean logic
  • Aggregations — compute statistics, histograms, and groupings in real-time
  • Near real-time — indexed data is searchable within ~1 second
  • Distributed — data is sharded across multiple nodes for scalability
  • RESTful API — all operations via HTTP/JSON (no special client needed)
  • Schema-free — can index JSON documents without predefined schema (but mappings are recommended)

Typical interaction:

# Index a document
$ curl -X POST "localhost:9200/my-index/_doc" \
  -H "Content-Type: application/json" \
  -d '{"message": "Hello ELK", "level": "info"}'

# Search for documents
$ curl -X GET "localhost:9200/my-index/_search?q=level:info"

Logstash

Role: Collect, transform, and route data.

Logstash is a server-side data processing pipeline. It ingests data from multiple sources, transforms it using filters, and sends it to one or more destinations.

Key features:

  • 200+ plugins — inputs, filters, and outputs for almost any data source
  • Grok parsing — parse unstructured log lines into structured fields
  • Conditional routing — send different log types to different indices
  • Enrichment — add GeoIP data, DNS lookups, or custom fields
  • Buffering — persistent queues to handle backpressure

A Logstash pipeline has three sections:

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "web-logs-%{+YYYY.MM.dd}"
  }
}

Kibana

Role: Visualize, explore, and manage data.

Kibana is the web UI for the Elastic Stack. It’s where you search logs, build dashboards, configure alerts, and manage your cluster.

Key features:

Feature Description
Discover Search and explore raw log events with filtering
Lens Drag-and-drop visualization builder
Dashboards Combine multiple visualizations on a single page
Canvas Pixel-perfect, presentation-ready reports
Maps Geographic visualizations with GeoIP data
Alerting Define rules and get notified (email, Slack, PagerDuty)
Dev Tools Console for running Elasticsearch queries directly
Stack Monitoring Monitor the health of your Elasticsearch cluster
Spaces Multi-tenant dashboards for different teams

Beats Family

Role: Lightweight data shippers installed on your servers.

Each Beat is purpose-built for a specific type of data:

Beat What It Collects Use Case
Filebeat Log files Tail log files and ship lines
Metricbeat System and service metrics CPU, memory, disk, network, Docker, K8s
Heartbeat Uptime monitoring Ping HTTP, TCP, ICMP endpoints
Packetbeat Network packets DNS, HTTP, MySQL, Redis protocol analysis
Auditbeat Audit events File integrity, system calls, user activity
Winlogbeat Windows Event Logs Windows security, application, system logs

Filebeat is the most commonly used Beat. It’s installed on every server that produces logs.

Why Beats instead of Logstash?

Aspect Beats Logstash
Weight Lightweight (~30 MB) Heavy (~500 MB, JVM)
CPU usage Minimal Moderate to high
Processing Basic (modules, processors) Advanced (grok, conditionals)
Where it runs On every server On dedicated servers
Scalability One per server Centralized

Best practice: Use Beats for collection and Logstash for heavy processing.

Elastic Agent & Fleet

Elastic Agent is the newer, unified approach to data collection. Instead of installing Filebeat + Metricbeat + Auditbeat separately, you install a single Elastic Agent that handles all data types.

┌──────────────────────────────────────────────┐
│              Traditional Approach              │
│                                               │
│  Server 1:  Filebeat + Metricbeat + Auditbeat│
│  Server 2:  Filebeat + Metricbeat + Auditbeat│
│  Server 3:  Filebeat + Metricbeat + Auditbeat│
│  (3 agents per server × 100 servers = 300)   │
└──────────────────────────────────────────────┘

┌──────────────────────────────────────────────┐
│              Elastic Agent Approach            │
│                                               │
│  Server 1:  Elastic Agent (handles all)      │
│  Server 2:  Elastic Agent (handles all)      │
│  Server 3:  Elastic Agent (handles all)      │
│  (1 agent per server × 100 servers = 100)    │
└──────────────────────────────────────────────┘

Fleet is the centralized management UI in Kibana for Elastic Agents:

  • Deploy agents to thousands of servers
  • Update configurations remotely
  • Apply integrations (NGINX, MySQL, Kubernetes, AWS, etc.)
  • Monitor agent health and status

Ingest Pipelines

Ingest Pipelines are an Elasticsearch-native alternative to Logstash for simple transformations. They run inside Elasticsearch itself — no separate service needed.

When to use Ingest Pipelines vs Logstash:

Feature Ingest Pipelines Logstash
Where it runs Inside Elasticsearch Separate server
Complexity Simple transforms Complex parsing
Grok support ✅ Yes ✅ Yes
Conditional logic Limited Full if/else
Multiple outputs ❌ No (ES only) ✅ Yes (Kafka, S3, etc.)
Performance impact Uses ES resources Dedicated resources

Example ingest pipeline:

{
  "description": "Parse NGINX logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{COMBINEDAPACHELOG}"]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["dd/MMM/yyyy:HH:mm:ss Z"]
      }
    }
  ]
}

Elasticsearch Clients & Integrations

Elasticsearch provides official clients for popular programming languages:

Language Client Library
Python elasticsearch-py
Java elasticsearch-java
JavaScript/Node.js @elastic/elasticsearch
Go go-elasticsearch
Ruby elasticsearch-ruby
PHP elasticsearch-php
.NET Elastic.Clients.Elasticsearch

Component Summary

┌─────────────────────────────────────────────────────┐
│               When to Use What                       │
│                                                      │
│  Collecting logs?        → Filebeat                  │
│  Collecting metrics?     → Metricbeat                │
│  Monitoring uptime?      → Heartbeat                 │
│  One agent for all?      → Elastic Agent + Fleet     │
│  Complex log parsing?    → Logstash                  │
│  Simple transforms?      → Ingest Pipelines          │
│  Storing & searching?    → Elasticsearch             │
│  Building dashboards?    → Kibana                    │
│  Running queries?        → Kibana Dev Tools          │
│  Managing agents?        → Fleet (in Kibana)         │
└─────────────────────────────────────────────────────┘

Quick Reference

Component Role Default Port
Elasticsearch Store and search 9200
Logstash Process and transform 5044
Kibana Visualize and manage 5601
Filebeat Ship log files —
Metricbeat Ship metrics —
Elastic Agent Unified data shipper —
Fleet Agent management (in Kibana) 5601
Ingest Pipelines ES-native transforms 9200

Next Step

Now that you know all the components, let’s see how ELK compares to alternatives like Splunk, Grafana Loki, and Datadog.

You may also enjoy