ELK Stack — Multiline Logs

The Multiline Problem

Most log parsing tools treat each line as a separate event. But many log entries span multiple lines:

• Java stack traces — exception + dozens of at ... lines
• Python tracebacks — Traceback + frame lines + exception
• XML/JSON blocks — pretty-printed payloads
• SQL queries — multi-line formatted queries

Without multiline handling, a single stack trace becomes dozens of broken events:

                     Without Multiline Handling
 ┌─────────────────────────────────────────────────────────┐
 │  Log File                    Elasticsearch              │
 │                                                         │
 │  ERROR NullPointer     →  Event 1: "ERROR NullPointer"  │
 │    at com.App.run()    →  Event 2: "at com.App.run()"   │
 │    at com.App.main()   →  Event 3: "at com.App.main()"  │
 │  Caused by: IOEx       →  Event 4: "Caused by: IOEx"    │
 │    at java.io.File()   →  Event 5: "at java.io.File()"  │
 └─────────────────────────────────────────────────────────┘

                      With Multiline Handling
 ┌─────────────────────────────────────────────────────────┐
 │  Log File                    Elasticsearch              │
 │                                                         │
 │  ERROR NullPointer     →  Event 1: "ERROR NullPointer   │
 │    at com.App.run()    →    at com.App.run()             │
 │    at com.App.main()   →    at com.App.main()            │
 │  Caused by: IOEx       →  Caused by: IOEx               │
 │    at java.io.File()   →    at java.io.File()"          │
 └─────────────────────────────────────────────────────────┘

Best practice: Handle multiline merging at the shipper level (Filebeat), before logs reach Logstash or Elasticsearch.

---

Filebeat Multiline Configuration

Filebeat’s multiline support is configured under filebeat.inputs:

filebeat.inputs:
  - type: filestream
    id: java-app-logs
    paths:
      - /var/log/myapp/*.log
    multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2}'
      negate: true
      match: after
      max_lines: 500
      timeout: 5s

Configuration Options

Option	Description
pattern	Regex pattern to identify the start (or continuation) of a multiline event
negate	true = lines NOT matching the pattern are continuation lines; false = lines matching are continuation lines
match	after = continuation lines go after the previous line; before = continuation lines go before the next line
max_lines	Maximum number of lines to merge (default: 500)
timeout	Maximum wait time for the next line before flushing (default: 5s)

Understanding negate + match

negate	match	Behavior
false	after	Lines matching the pattern are appended to the previous event
false	before	Lines matching the pattern are prepended to the next event
true	after	Lines not matching the pattern are appended to the previous event
true	before	Lines not matching the pattern are prepended to the next event

The most common combination is negate: true + match: after — which means “every line that does NOT start a new log event gets appended to the previous event.”

---

Java Stack Trace — Complete Example

Raw log:

2026-04-01 10:15:30.456 [http-nio-8080-exec-1] ERROR c.e.UserController - Failed to process request
java.lang.NullPointerException: User not found
	at com.example.service.UserService.getUser(UserService.java:42)
	at com.example.controller.UserController.handleRequest(UserController.java:28)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
Caused by: java.sql.SQLException: Connection refused
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:800)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:407)
	... 15 more
2026-04-01 10:15:31.123 [http-nio-8080-exec-2] INFO  c.e.HealthController - Health check OK

Filebeat Config

Java log lines typically start with a timestamp YYYY-MM-DD:

filebeat.inputs:
  - type: filestream
    id: java-app-logs
    paths:
      - /var/log/myapp/*.log
    multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2}'
      negate: true
      match: after
      max_lines: 500
      timeout: 5s

Logic: Any line NOT starting with YYYY-MM-DD is a continuation of the previous event.

Result: The entire stack trace is merged into a single event:

{
  "message": "2026-04-01 10:15:30.456 [http-nio-8080-exec-1] ERROR c.e.UserController - Failed to process request\njava.lang.NullPointerException: User not found\n\tat com.example.service.UserService.getUser(UserService.java:42)\n\tat com.example.controller.UserController.handleRequest(UserController.java:28)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\nCaused by: java.sql.SQLException: Connection refused\n\tat com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:800)\n\tat com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:407)\n\t... 15 more"
}

Logstash Parsing (after Filebeat merges)

filter {
  grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:thread}\] %{LOGLEVEL:level}\s+%{NOTSPACE:logger} - %{GREEDYDATA:log_message}"
    }
  }

  # Extract exception class from the stack trace
  if [log_message] =~ /\n/ {
    grok {
      match => {
        "log_message" => "(?m)%{DATA:error_summary}\n%{NOTSPACE:exception_class}: %{DATA:exception_message}\n%{GREEDYDATA:stack_trace}"
      }
      tag_on_failure => ["_grok_stacktrace_fail"]
    }
  }
}

---

Python Traceback — Complete Example

Raw log:

2026-04-01 10:15:30 ERROR app.views - Unhandled exception
Traceback (most recent call last):
  File "/app/views.py", line 42, in get_user
    user = User.objects.get(pk=user_id)
  File "/app/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in get
    return self.get_queryset().get(*args, **kwargs)
app.models.User.DoesNotExist: User matching query does not exist.
2026-04-01 10:15:31 INFO  app.middleware - Request completed

Filebeat Config

Python log lines typically start with a timestamp:

filebeat.inputs:
  - type: filestream
    id: python-app-logs
    paths:
      - /var/log/python-app/*.log
    multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2}'
      negate: true
      match: after
      max_lines: 200
      timeout: 5s

Alternative: Match on Traceback Start

If your Python logs don’t always start with a timestamp, match on Traceback:

filebeat.inputs:
  - type: filestream
    id: python-app-logs
    paths:
      - /var/log/python-app/*.log
    multiline:
      type: pattern
      pattern: '^Traceback|^  File|^\s+|^[A-Za-z_][A-Za-z0-9_.]*Error|^[A-Za-z_][A-Za-z0-9_.]*Exception'
      negate: false
      match: after
      max_lines: 200
      timeout: 5s

---

Logstash Multiline Codec

If you cannot use Filebeat multiline, Logstash has a multiline codec for input plugins:

input {
  file {
    path => "/var/log/myapp/*.log"
    start_position => "beginning"
    codec => multiline {
      pattern => "^\d{4}-\d{2}-\d{2}"
      negate => true
      what => "previous"
      max_lines => 500
      max_bytes => "10 MiB"
    }
  }
}

Logstash Codec	Filebeat Equivalent
what => "previous"	match: after
what => "next"	match: before

⚠️ Warning: Using the multiline codec in Logstash means multiline merging happens on the Logstash node, not the shipper. This consumes more Logstash memory and can cause issues with multiple workers. Prefer Filebeat multiline whenever possible.

---

Logstash Multiline Filter

The multiline filter plugin is deprecated in favor of the multiline codec or Filebeat multiline. If you still need it:

filter {
  multiline {
    pattern => "^\s"
    what => "previous"
    negate => false
  }
}

⚠️ Deprecated: The multiline filter does not work with multiple Logstash pipeline workers (-w). Use the multiline codec or Filebeat multiline instead.

---

Common Multiline Patterns

Java Stack Traces

# Lines start with timestamp (YYYY-MM-DD or ISO 8601)
multiline:
  type: pattern
  pattern: '^\d{4}-\d{2}-\d{2}[T ]'
  negate: true
  match: after

Python Tracebacks

# Lines start with timestamp
multiline:
  type: pattern
  pattern: '^\d{4}-\d{2}-\d{2}'
  negate: true
  match: after

XML/JSON Blocks

# JSON blocks start with {
multiline:
  type: pattern
  pattern: '^\{'
  negate: true
  match: after

SQL Queries (continuation lines start with whitespace)

multiline:
  type: pattern
  pattern: '^\s'
  negate: false
  match: after

Log4j Pattern (lines start with log level or timestamp)

multiline:
  type: pattern
  pattern: '^(TRACE|DEBUG|INFO|WARN|ERROR|FATAL|\d{4})'
  negate: true
  match: after

---

Best Practices

Practice	Description
Handle at the shipper	Use Filebeat multiline, not Logstash multiline filter
Set max_lines	Prevent runaway merging — 500 is a reasonable default
Set timeout	5s is usually sufficient — prevents waiting forever for the next line
Test patterns thoroughly	A wrong pattern can merge unrelated events or split valid ones
Use structured logging	JSON logs avoid the multiline problem entirely
Monitor parse failures	Track _grokparsefailure tags for merged events that fail parsing
Consider log format	If you control the app, add a unique prefix to every log line

---

Quick Reference

# Filebeat multiline (recommended)
multiline:
  type: pattern
  pattern: '^\d{4}-\d{2}-\d{2}'   # Regex for line start
  negate: true                      # Lines NOT matching = continuation
  match: after                      # Append to previous event
  max_lines: 500                    # Safety limit
  timeout: 5s                       # Flush timeout

# Logstash multiline codec (if Filebeat not used)
codec => multiline {
  pattern => "^\d{4}-\d{2}-\d{2}"
  negate => true
  what => "previous"
  max_lines => 500
}

# Common negate + match combinations
negate: true  + match: after   → Most common (timestamp-prefixed logs)
negate: false + match: after   → Continuation lines match the pattern
negate: true  + match: before  → Rare — merge upward

---

Previous: Dissect Filter ← · Next: Structured Logging →