ELK Stack — Custom Grok Patterns

4 minute read

Why Custom Grok Patterns?

Built-in Grok patterns cover standard log formats like Apache, Syslog, and ISO timestamps. But real-world applications produce custom log formats that don’t match any built-in pattern. Custom Grok patterns let you:

  • Parse application-specific log formats
  • Create reusable pattern libraries shared across pipelines
  • Build complex patterns by composing simple building blocks
  • Keep Logstash configs readable by abstracting regex into named patterns

Creating Custom Pattern Files

Pattern File Format

Custom pattern files are plain text — one pattern per line:

PATTERN_NAME regex_definition

Example — /etc/logstash/patterns/custom:

# Application log patterns
APP_LOGLEVEL (TRACE|DEBUG|INFO|WARN|ERROR|FATAL)
APP_THREAD [a-zA-Z0-9._-]+
APP_CLASS [a-zA-Z0-9.$_]+
APP_TIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}:%{SECOND}(?:\.%{INT})?

# Request ID pattern (UUID format)
REQUEST_ID [0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}

# Duration in various formats
DURATION_MS %{NUMBER}ms
DURATION_SEC %{NUMBER}s

Using patterns_dir in Logstash

Point the Grok filter to your custom patterns directory:

filter {
  grok {
    patterns_dir => ["/etc/logstash/patterns"]
    match => {
      "message" => "%{APP_TIMESTAMP:timestamp} \[%{APP_THREAD:thread}\] %{APP_LOGLEVEL:level} %{APP_CLASS:logger} - %{GREEDYDATA:log_message}"
    }
  }
}

Multiple directories are supported:

filter {
  grok {
    patterns_dir => [
      "/etc/logstash/patterns",
      "/etc/logstash/patterns.d/app-specific"
    ]
    match => {
      "message" => "%{MY_APP_LOG}"
    }
  }
}

Named Captures

For one-off patterns that don’t need a pattern file, use named captures — inline regex with a field name:

(?<field_name>pattern)

Example:

filter {
  grok {
    match => {
      "message" => "(?<request_id>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) %{GREEDYDATA:log_message}"
    }
  }
}

Named captures are useful when:

  • The pattern is used only once
  • You need a quick inline regex
  • The regex is too specific for a reusable pattern

Composing Complex Patterns

Build complex patterns by composing simple ones. Start small, test each piece, then combine.

Step-by-Step: Parsing a Custom Log Format

Target log format:

[2026-04-01T10:15:30.456Z] [req-a1b2c3d4-e5f6-7890-abcd-ef1234567890] [INFO] [UserService] POST /api/users 201 45ms - Created user john@example.com

Step 1 — Break down the format:

[timestamp] [request_id] [level] [service] method path status duration - message

Step 2 — Create patterns for each component:

# File: /etc/logstash/patterns/myapp

# Timestamp with milliseconds and timezone
MYAPP_TS %{TIMESTAMP_ISO8601}

# Request ID with prefix
MYAPP_REQ_ID req-[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}

# Log level
MYAPP_LEVEL (TRACE|DEBUG|INFO|WARN|ERROR|FATAL)

# Service name
MYAPP_SERVICE [A-Za-z][A-Za-z0-9_]*

# HTTP components
MYAPP_METHOD (GET|POST|PUT|DELETE|PATCH|HEAD|OPTIONS)
MYAPP_PATH /[^\s]*
MYAPP_STATUS [1-5][0-9]{2}
MYAPP_DURATION %{NUMBER}ms

Step 3 — Compose the full pattern:

# Composite pattern
MYAPP_LOG \[%{MYAPP_TS:timestamp}\] \[%{MYAPP_REQ_ID:request_id}\] \[%{MYAPP_LEVEL:level}\] \[%{MYAPP_SERVICE:service}\] %{MYAPP_METHOD:method} %{MYAPP_PATH:path} %{MYAPP_STATUS:status} %{MYAPP_DURATION:duration} - %{GREEDYDATA:log_message}

Step 4 — Use in Logstash:

filter {
  grok {
    patterns_dir => ["/etc/logstash/patterns"]
    match => {
      "message" => "%{MYAPP_LOG}"
    }
  }

  # Convert types
  mutate {
    convert => {
      "status" => "integer"
    }
    gsub => [
      "duration", "ms$", ""
    ]
  }
  mutate {
    convert => {
      "duration" => "integer"
    }
  }
}

Raw log:

[2026-04-01T10:15:30.456Z] [req-a1b2c3d4-e5f6-7890-abcd-ef1234567890] [INFO] [UserService] POST /api/users 201 45ms - Created user john@example.com

Parsed output:

{
  "timestamp": "2026-04-01T10:15:30.456Z",
  "request_id": "req-a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "level": "INFO",
  "service": "UserService",
  "method": "POST",
  "path": "/api/users",
  "status": 201,
  "duration": 45,
  "log_message": "Created user john@example.com"
}

Custom Patterns for JSON-Embedded Logs

Some logs embed JSON within a text prefix:

2026-04-01 10:15:30 INFO  AppServer: {"event":"login","user":"admin","ip":"10.0.0.5","success":true}

Parse the text prefix with Grok, then use the json filter for the JSON body:

filter {
  grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}\s+%{WORD:source}: %{GREEDYDATA:json_body}"
    }
  }

  json {
    source => "json_body"
    target => "event_data"
  }

  mutate {
    remove_field => ["json_body"]
  }
}

Parsed output:

{
  "timestamp": "2026-04-01 10:15:30",
  "level": "INFO",
  "source": "AppServer",
  "event_data": {
    "event": "login",
    "user": "admin",
    "ip": "10.0.0.5",
    "success": true
  }
}

Custom Patterns for Multi-Format Logs

When a single source produces logs in different formats, create a pattern for each and use multiple match patterns:

# File: /etc/logstash/patterns/multiformat

# Format A: API request logs
FORMAT_API %{TIMESTAMP_ISO8601:timestamp} \[API\] %{WORD:method} %{URIPATH:path} %{NUMBER:status:int} %{NUMBER:duration_ms:int}ms
# Format B: Database query logs
FORMAT_DB %{TIMESTAMP_ISO8601:timestamp} \[DB\] %{WORD:operation} %{WORD:table} %{NUMBER:rows:int} rows %{NUMBER:duration_ms:int}ms
# Format C: Error logs
FORMAT_ERR %{TIMESTAMP_ISO8601:timestamp} \[ERROR\] %{NOTSPACE:error_code} %{GREEDYDATA:error_message}

filter {
  grok {
    patterns_dir => ["/etc/logstash/patterns"]
    match => {
      "message" => [
        "%{FORMAT_API}",
        "%{FORMAT_DB}",
        "%{FORMAT_ERR}"
      ]
    }
    break_on_match => true
  }

  # Add a log_type field based on which pattern matched
  if [method] {
    mutate { add_field => { "log_type" => "api" } }
  } else if [operation] {
    mutate { add_field => { "log_type" => "database" } }
  } else if [error_code] {
    mutate { add_field => { "log_type" => "error" } }
  }
}

Testing Custom Patterns

Using Grok Debugger in Kibana

  1. Go to Dev Tools → Grok Debugger
  2. Enter custom patterns in the Custom Patterns section
  3. Paste your sample log
  4. Enter the composite Grok pattern
  5. Click Simulate

Using Logstash in Debug Mode

Create a minimal test pipeline:

# test-grok.conf
input {
  stdin {}
}

filter {
  grok {
    patterns_dir => ["/etc/logstash/patterns"]
    match => {
      "message" => "%{MYAPP_LOG}"
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Run and paste test log lines:

echo '[2026-04-01T10:15:30.456Z] [req-a1b2c3d4-e5f6-7890-abcd-ef1234567890] [INFO] [UserService] POST /api/users 201 45ms - Created user' | /usr/share/logstash/bin/logstash -f test-grok.conf

Best Practices

Practice Description
One pattern per concern Keep patterns focused — don’t put everything in one pattern
Use descriptive names MYAPP_REQ_ID is better than RID
Test incrementally Build and test one component at a time
Document patterns Add comments in pattern files explaining the format
Version pattern files Store pattern files in source control alongside configs
Avoid overly broad regex GREEDYDATA should only be used at the end of a pattern
Namespace your patterns Prefix with app name: MYAPP_*, BILLING_*

Quick Reference

# Pattern file format (one per line)
PATTERN_NAME regex_or_composed_pattern

# Using custom patterns directory
filter {
  grok {
    patterns_dir => ["/etc/logstash/patterns"]
    match => { "message" => "%{MY_PATTERN:field}" }
  }
}

# Named capture (inline regex)
(?<field_name>regex)

# Composing patterns
SIMPLE_A [0-9]+
SIMPLE_B [a-zA-Z]+
COMPOSED %{SIMPLE_A:num} %{SIMPLE_B:word}

# Multi-format matching
match => {
  "message" => [
    "%{FORMAT_A}",
    "%{FORMAT_B}",
    "%{FORMAT_C}"
  ]
}
break_on_match => true

Previous: Grok Patterns ← · Next: Dissect Filter →

You may also enjoy