ELK Stack — Logstash

2 minute read

Logstash — The Data Processing Pipeline

Logstash is a powerful, open-source data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and sends it to your desired destination. Think of Logstash as the ETL engine (Extract, Transform, Load) of the Elastic Stack.

While Beats are lightweight shippers that forward raw data, Logstash is the heavy lifter — it can parse, enrich, filter, and restructure data before it reaches Elasticsearch.

┌─────────────┐    ┌────────────────────────────────┐    ┌───────────────┐
│   INPUTS    │───▶│         LOGSTASH               │───▶│   OUTPUTS     │
│             │    │  ┌────────┐    ┌────────────┐   │    │               │
│ • Beats     │    │  │ Filter │───▶│  Transform │   │    │ • Elasticsearch│
│ • Syslog    │    │  └────────┘    └────────────┘   │    │ • S3          │
│ • Kafka     │    │                                  │    │ • Kafka       │
│ • Files     │    │    Parse • Enrich • Structure    │    │ • Files       │
└─────────────┘    └────────────────────────────────┘    └───────────────┘

What You’ll Learn

In this chapter, you will learn how to:

  • 🔧 Understand pipeline architecture — inputs, filters, and outputs working together
  • 📥 Configure input plugins — ingest data from files, Beats, Kafka, databases, and more
  • 🔄 Apply filter plugins — parse, transform, and enrich log data using grok, mutate, date, and other filters
  • 📤 Set up output plugins — send processed data to Elasticsearch, files, Kafka, and other destinations
  • 🔀 Run multiple pipelines — isolate data flows, configure pipeline-to-pipeline communication, and monitor performance
  • ✅ Practice with hands-on exercises — build real-world pipelines from scratch

Chapter Contents

# Topic Description
1 Pipeline Architecture Logstash pipeline concept, config file structure, event lifecycle
2 Input Plugins Ingest data from files, Beats, Kafka, syslog, databases
3 Filter Plugins Parse and transform data with grok, mutate, date, geoip
4 Output Plugins Send data to Elasticsearch, files, Kafka, and more
5 Multiple Pipelines Run isolated pipelines, pipeline-to-pipeline, DLQ
6 Practice Exercises Hands-on exercises from basic to advanced

Why Logstash?

Feature Description
200+ Plugins Inputs, filters, and outputs for virtually any data source or destination
Real-time Processing Process events as they arrive with low latency
Extensible Write custom plugins in Ruby or use the Ruby filter for inline logic
Reliable Persistent queues, dead letter queues, and at-least-once delivery
Scalable Horizontal scaling with multiple pipeline workers and instances

When to Use Logstash vs. Beats

Use Case Recommended
Simple log forwarding Beats
Complex parsing / transformation Logstash
Lightweight edge collection Beats
Multi-source aggregation Logstash
Enrichment (GeoIP, DNS, etc.) Logstash
Beats → Elasticsearch (no transform) Beats only
Beats → Transform → Elasticsearch Beats + Logstash

Tip: In production, the most common pattern is Beats → Logstash → Elasticsearch. Beats handle lightweight collection on each host, and Logstash centralizes parsing and enrichment.

Let’s start by understanding how the Logstash pipeline works.

👉 Next: Pipeline Architecture

You may also enjoy