ELK Stack — Install Logstash

3 minute read

Install Logstash

Logstash is the data processing engine of the ELK Stack. It ingests data from multiple sources, transforms it, and sends it to Elasticsearch. Let’s install it and test a basic pipeline.

Install on Ubuntu/Debian

$ sudo apt install logstash -y

Install on CentOS/RHEL

$ sudo yum install logstash -y

Install on macOS (Homebrew)

$ brew install elastic/tap/logstash-full

Verify Installation

$ /usr/share/logstash/bin/logstash --version

Expected output:

logstash 8.15.0

Understanding Logstash Configuration

Every Logstash pipeline has three sections:

┌──────────────────────────────────────────────┐
│              Logstash Pipeline                │
│                                               │
│  input {                                      │
│    # WHERE to get data from                  │
│  }                                            │
│                                               │
│  filter {                                     │
│    # HOW to process/transform data           │
│  }                                            │
│                                               │
│  output {                                     │
│    # WHERE to send processed data            │
│  }                                            │
└──────────────────────────────────────────────┘

Test 1 — Hello World Pipeline

Let’s start with the simplest possible pipeline — read from stdin and write to stdout:

Create a test pipeline file:

$ sudo nano /etc/logstash/conf.d/test-stdin.conf

input {
  stdin {}
}

output {
  stdout {
    codec => rubydebug
  }
}

Run it:

$ /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test-stdin.conf

Wait for Logstash to start (it takes 30-60 seconds), then type:

Hello ELK Stack

Expected output:

{
    "@timestamp" => 2025-03-15T10:00:00.000Z,
      "@version" => "1",
       "message" => "Hello ELK Stack",
          "host" => {
        "hostname" => "my-server"
    }
}

Logstash wrapped your text in a structured event with a timestamp and metadata. Press Ctrl+C to stop.

Test 2 — Pipeline with a Filter

Create a pipeline that parses Apache-style log lines:

$ sudo nano /etc/logstash/conf.d/test-grok.conf

input {
  stdin {}
}

filter {
  grok {
    match => { "message" => "%{IP:client_ip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}\" %{NUMBER:status_code:int} %{NUMBER:bytes:int}" }
  }

  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    target => "@timestamp"
  }

  mutate {
    remove_field => [ "message", "timestamp" ]
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Run it:

$ /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test-grok.conf

Paste this sample log line:

192.168.1.50 - - [15/Mar/2025:10:15:30 +0000] "GET /api/users HTTP/1.1" 200 1234

Expected output:

{
      "client_ip" => "192.168.1.50",
         "method" => "GET",
           "path" => "/api/users",
   "http_version" => "1.1",
    "status_code" => 200,
          "bytes" => 1234,
     "@timestamp" => 2025-03-15T10:15:30.000Z,
       "@version" => "1"
}

Logstash parsed the unstructured log line into individual fields. This is the power of Logstash filters.

Test 3 — Send to Elasticsearch

Create a pipeline that reads from Beats and sends to Elasticsearch:

$ sudo nano /etc/logstash/conf.d/beats-to-es.conf

input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][log_type] == "nginx" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => "elastic"
    password => "YOUR_PASSWORD"
    ssl_certificate_verification => false
    index => "logs-%{+YYYY.MM.dd}"
  }
}

Replace YOUR_PASSWORD with your Elasticsearch password.

Configure Logstash Settings

Edit the main Logstash settings:

$ sudo nano /etc/logstash/logstash.yml

Key settings:

# Pipeline settings
pipeline.workers: 2
pipeline.batch.size: 125
pipeline.batch.delay: 50

# Path settings
path.data: /var/lib/logstash
path.logs: /var/log/logstash

# Pipeline configuration directory
path.config: /etc/logstash/conf.d

# Monitoring (optional)
monitoring.enabled: false

Configure JVM Settings

$ sudo nano /etc/logstash/jvm.options

Adjust heap size:

-Xms1g
-Xmx1g

For learning, 512 MB to 1 GB is sufficient.

Start Logstash as a Service

$ sudo systemctl daemon-reload
$ sudo systemctl enable logstash
$ sudo systemctl start logstash

Check status:

$ sudo systemctl status logstash

Check Logstash Logs

$ sudo tail -f /var/log/logstash/logstash-plain.log

Look for:

[INFO ] Pipelines running {:count=>1, :running_pipelines=>[:main]}
[INFO ] Successfully started Logstash API endpoint {:port=>9600}

Test the Logstash API

$ curl -s localhost:9600 | python3 -m json.tool

Expected output:

{
  "host": "my-server",
  "version": "8.15.0",
  "http_address": "127.0.0.1:9600",
  "id": "...",
  "name": "my-server",
  "status": "green",
  "pipeline": {
    "workers": 2,
    "batch_size": 125,
    "batch_delay": 50
  }
}

Important File Locations

File Path
Configuration /etc/logstash/logstash.yml
Pipeline configs /etc/logstash/conf.d/
JVM options /etc/logstash/jvm.options
Data directory /var/lib/logstash
Log directory /var/log/logstash
Binaries /usr/share/logstash/bin/
Plugins /usr/share/logstash/

Common Issues

Issue Solution
Logstash takes long to start Normal — JVM startup takes 30-60 seconds
“Could not find any pipeline” Check path.config points to your .conf files
Grok parse failure Test your grok pattern at grokdebugger.com
Out of memory Reduce heap in jvm.options
Pipeline error in logs Check /var/log/logstash/logstash-plain.log
Port 5044 already in use Another instance may be running: sudo lsof -i :5044

Quick Reference

Task Command
Start Logstash sudo systemctl start logstash
Stop Logstash sudo systemctl stop logstash
Check status sudo systemctl status logstash
View logs sudo tail -f /var/log/logstash/logstash-plain.log
Test pipeline /usr/share/logstash/bin/logstash -f <config>
Check API curl localhost:9600
Default ports Beats input: 5044, API: 9600

Next Step

Logstash is ready! Let’s now install Filebeat to ship logs from your servers.

You may also enjoy