ELK Stack — Pipeline Architecture

4 minute read

Pipeline Architecture

Every Logstash pipeline consists of three stages: inputs, filters, and outputs. Data flows through these stages as discrete units called events.

┌──────────┐    ┌─────────────────┐    ┌──────────┐    ┌──────────┐
│  INPUT   │───▶│  INTERNAL QUEUE │───▶│  FILTER  │───▶│  OUTPUT  │
│          │    │  (in-memory or  │    │          │    │          │
│ • beats  │    │   persistent)   │    │ • grok   │    │ • ES     │
│ • file   │    │                 │    │ • mutate │    │ • file   │
│ • stdin  │    │                 │    │ • date   │    │ • stdout │
└──────────┘    └─────────────────┘    └──────────┘    └──────────┘
                                            │
                                       ┌────┴────┐
                                       │  CODEC  │
                                       │(decode/ │
                                       │ encode) │
                                       └─────────┘

How It Works

  1. Input — generates events. An input plugin reads data from a source (file, network, queue) and creates an event.
  2. Queue — buffers events between input and filter stages. Can be in-memory (default) or persistent (disk-backed).
  3. Filter — processes events. Filter plugins parse, transform, and enrich data. Filters are optional but almost always used.
  4. Output — ships events. An output plugin sends the processed event to a destination (Elasticsearch, file, Kafka, etc.).
  5. Codec — acts as a stream decoder/encoder. Codecs operate on the input or output stage to decode/encode data formats (JSON, multiline, plain, rubydebug).

Pipeline Configuration File

A Logstash pipeline is defined in a .conf file with three sections:

input {
  # Define data sources here
}

filter {
  # Transform and enrich data here
}

output {
  # Define destinations here
}

Each section can contain one or more plugin configurations.

Your First Pipeline — stdin to stdout

The simplest pipeline reads from the terminal and prints back to it:

# first-pipeline.conf
input {
  stdin { }
}

output {
  stdout {
    codec => rubydebug
  }
}

Run the pipeline:

bin/logstash -f first-pipeline.conf

Type hello world and press Enter. You’ll see:

{
    "@timestamp" => 2026-04-01T09:06:00.000Z,
       "message" => "hello world",
      "@version" => "1",
          "host" => {
        "hostname" => "logstash-server"
    },
         "event" => {
        "original" => "hello world"
    }
}

Every event automatically gets @timestamp, @version, host, and event.original fields.

Adding a Filter

Let’s add a filter to parse a comma-separated line:

# csv-pipeline.conf
input {
  stdin { }
}

filter {
  csv {
    separator => ","
    columns => ["name", "age", "city"]
  }

  mutate {
    convert => { "age" => "integer" }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Sample input:

Alice,30,New York

Expected output:

{
    "@timestamp" => 2026-04-01T09:10:00.000Z,
       "message" => "Alice,30,New York",
          "name" => "Alice",
           "age" => 30,
          "city" => "New York",
      "@version" => "1"
}

Running Logstash with Config File

# Run with a single config file
bin/logstash -f /etc/logstash/conf.d/my-pipeline.conf

# Run with a directory of config files (merged into one pipeline)
bin/logstash -f /etc/logstash/conf.d/

# Test configuration syntax without starting
bin/logstash --config.test_and_exit -f my-pipeline.conf

# Auto-reload config on changes
bin/logstash -f my-pipeline.conf --config.reload.automatic

logstash.yml — Global Settings

The logstash.yml file controls global Logstash behavior:

# /etc/logstash/logstash.yml

# Node identity
node.name: "logstash-prod-01"

# Pipeline settings
pipeline.workers: 4              # Number of filter/output workers (default: CPU cores)
pipeline.batch.size: 125         # Events per worker batch
pipeline.batch.delay: 50         # Milliseconds to wait for batch to fill
pipeline.ordered: auto           # Event ordering (auto, true, false)

# Config path
path.config: /etc/logstash/conf.d

# Data and logs
path.data: /var/lib/logstash
path.logs: /var/log/logstash

# Auto-reload
config.reload.automatic: true
config.reload.interval: 3s

# Persistent queue
queue.type: memory               # memory (default) or persisted
queue.max_bytes: 1024mb          # Max queue size (persisted only)

# Dead letter queue
dead_letter_queue.enable: false
dead_letter_queue.max_bytes: 1024mb

# HTTP API
api.http.host: "127.0.0.1"
api.http.port: 9600

# Monitoring
monitoring.enabled: true

pipelines.yml — Multiple Pipelines

Instead of merging all configs into one pipeline, you can run multiple isolated pipelines:

# /etc/logstash/pipelines.yml

- pipeline.id: apache-logs
  path.config: "/etc/logstash/conf.d/apache.conf"
  pipeline.workers: 2
  pipeline.batch.size: 250

- pipeline.id: application-logs
  path.config: "/etc/logstash/conf.d/app.conf"
  pipeline.workers: 4

- pipeline.id: metrics
  path.config: "/etc/logstash/conf.d/metrics.conf"
  pipeline.workers: 1

Each pipeline runs independently with its own worker threads.

Event Lifecycle

Every event that flows through Logstash follows this lifecycle:

1. INPUT creates event with raw data
       ↓
2. CODEC decodes the raw data (e.g., JSON, plain, multiline)
       ↓
3. Event enters the QUEUE (in-memory or persisted)
       ↓
4. Worker thread picks up event from queue
       ↓
5. FILTERS process the event sequentially
   • grok parses unstructured text
   • mutate modifies fields
   • date sets @timestamp
   • conditionals route processing
       ↓
6. OUTPUT sends the event to destination(s)
       ↓
7. CODEC encodes the event for output (e.g., rubydebug, json_lines)
       ↓
8. Event is acknowledged and removed from queue

Codecs — Decoding and Encoding

Codecs handle serialization on the input/output boundary:

Codec Use Case Example
plain Raw text lines (default for stdin) Each line is one event
json JSON-formatted input/output Parse JSON automatically
json_lines Newline-delimited JSON One JSON object per line
multiline Multi-line events (stack traces) Merge lines into one event
rubydebug Pretty-print events (output only) Human-readable debug output
line Single line per event (output) Output events as plain lines

Multiline codec example (Java stack traces):

input {
  file {
    path => "/var/log/app/application.log"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
    }
  }
}

This merges lines that do NOT start with a timestamp into the previous event — perfect for stack traces.

Quick Reference

Concept Description
Pipeline Input → Filter → Output processing chain
Event A single unit of data flowing through the pipeline
Codec Encodes/decodes data at input or output boundary
Worker Thread that executes filter and output stages
.conf file Pipeline configuration (input/filter/output blocks)
logstash.yml Global Logstash settings
pipelines.yml Defines multiple independent pipelines
--config.test_and_exit Validates config syntax without running
--config.reload.automatic Auto-reload config on file changes

Tip: Always test your pipeline configuration with --config.test_and_exit before deploying to production.

👉 Next: Input Plugins

You may also enjoy