ELK Stack — Grok Patterns

5 minute read

What Is Grok?

Grok is a pattern-matching syntax built on top of regular expressions. It ships with over 120 pre-built patterns for common log formats. Instead of writing raw regex, you compose readable patterns from named building blocks.

Grok is available in:

  • Logstash — grok filter plugin
  • Elasticsearch — grok ingest processor
  • Kibana — Grok Debugger (Dev Tools)

Grok Syntax

The basic Grok syntax is:

%{PATTERN:field_name}
%{PATTERN:field_name:data_type}
Component Description
PATTERN A named regex pattern (built-in or custom)
field_name The field name to assign the matched value to
data_type Optional type conversion — int, float, or boolean

Examples:

%{IP:client_ip}              → Matches an IP address, stores as "client_ip"
%{NUMBER:bytes:int}          → Matches a number, converts to integer
%{WORD:http_method}          → Matches a single word
%{GREEDYDATA:message}        → Matches everything remaining

Built-In Grok Patterns

Grok ships with a rich library of pre-built patterns. Here are the most commonly used ones:

General Patterns

Pattern Matches Example
WORD A single word \b\w+\b GET, error, admin
NUMBER Integer or decimal 200, 3.14, -42
INT Integer only 200, -42
DATA Non-greedy match .*? Anything (shortest match)
GREEDYDATA Greedy match .* Anything (longest match)
NOTSPACE Non-whitespace characters 192.168.1.1, /api/users
SPACE Whitespace Spaces, tabs
QS Quoted string "GET /api HTTP/1.1"
QUOTEDSTRING Same as QS "some value"

Network Patterns

Pattern Matches Example
IP IPv4 or IPv6 192.168.1.1, ::1
IPV4 IPv4 only 10.0.0.1
IPV6 IPv6 only fe80::1
HOSTNAME Hostname or IP web01.example.com
HOSTPORT Host with port web01:8080
IPORHOST IP or hostname 192.168.1.1 or web01
URI Full URI https://example.com/path?q=1
URIPATH URI path portion /api/v1/users

Date/Time Patterns

Pattern Matches Example
TIMESTAMP_ISO8601 ISO 8601 timestamp 2026-04-01T10:15:30+00:00
HTTPDATE Apache-style date 01/Apr/2026:10:15:30 +0000
DATE_US US date 04-01-2026, 04/01/2026
DATE_EU European date 01-04-2026, 01.04.2026
TIME Time HH:MM:SS 10:15:30

Composite Patterns (Log Formats)

Pattern Matches
COMBINEDAPACHELOG Full Apache combined access log line
COMMONAPACHELOG Apache common log format
SYSLOGBASE Syslog priority, timestamp, host, process
SYSLOG5424BASE RFC 5424 syslog header

Using Grok in Logstash

Basic Grok Filter

filter {
  grok {
    match => {
      "message" => "%{COMBINEDAPACHELOG}"
    }
  }
}

Multiple Match Patterns

When logs have different formats, provide multiple patterns. Logstash tries each pattern in order:

filter {
  grok {
    match => {
      "message" => [
        "%{COMBINEDAPACHELOG}",
        "%{COMMONAPACHELOG}",
        "%{SYSLOGBASE} %{GREEDYDATA:syslog_message}"
      ]
    }
    break_on_match => true
  }
}

The break_on_match option (default: true) stops after the first successful match. Set it to false to apply all patterns.

Type Conversion

filter {
  grok {
    match => {
      "message" => "%{IP:client_ip} - %{WORD:user} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATH:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:status:int} %{NUMBER:bytes:int}"
    }
  }
}

Tag on Failure

When no pattern matches, Logstash adds _grokparsefailure to the tags field. You can customize this:

filter {
  grok {
    match => {
      "message" => "%{COMBINEDAPACHELOG}"
    }
    tag_on_failure => ["_grok_apache_fail"]
  }
}

Using Grok in Elasticsearch Ingest Pipeline

PUT _ingest/pipeline/apache-access-log
{
  "description": "Parse Apache access logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{COMBINEDAPACHELOG}"
        ]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["dd/MMM/yyyy:HH:mm:ss Z"],
        "target_field": "@timestamp"
      }
    },
    {
      "convert": {
        "field": "response",
        "type": "integer"
      }
    },
    {
      "convert": {
        "field": "bytes",
        "type": "integer"
      }
    }
  ]
}

Test the pipeline:

POST _ingest/pipeline/apache-access-log/_simulate
{
  "docs": [
    {
      "_source": {
        "message": "192.168.1.50 - admin [01/Apr/2026:10:15:30 +0000] \"GET /api/users HTTP/1.1\" 200 1234 \"https://example.com/\" \"Mozilla/5.0\""
      }
    }
  ]
}

Example 1 — Parsing Apache Access Logs

Raw log:

192.168.1.50 - admin [01/Apr/2026:10:15:30 +0000] "GET /api/users HTTP/1.1" 200 1234 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)"

Grok pattern:

%{COMBINEDAPACHELOG}

Or equivalently, the expanded form:

%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

Parsed output:

{
  "clientip": "192.168.1.50",
  "ident": "-",
  "auth": "admin",
  "timestamp": "01/Apr/2026:10:15:30 +0000",
  "verb": "GET",
  "request": "/api/users",
  "httpversion": "1.1",
  "response": "200",
  "bytes": "1234",
  "referrer": "\"https://example.com/\"",
  "agent": "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)\""
}

Example 2 — Parsing Syslog

Raw log:

Apr  1 10:15:30 web01 sshd[12345]: Accepted publickey for admin from 10.0.0.5 port 52413 ssh2

Grok pattern:

%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:process}\[%{POSINT:pid:int}\]: %{GREEDYDATA:syslog_message}

Logstash config:

filter {
  grok {
    match => {
      "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:process}\[%{POSINT:pid:int}\]: %{GREEDYDATA:syslog_message}"
    }
  }
}

Parsed output:

{
  "syslog_timestamp": "Apr  1 10:15:30",
  "hostname": "web01",
  "process": "sshd",
  "pid": 12345,
  "syslog_message": "Accepted publickey for admin from 10.0.0.5 port 52413 ssh2"
}

Example 3 — Parsing Custom Application Logs

Raw log:

2026-04-01 10:15:30.456 [main] ERROR com.example.UserService - Failed to fetch user id=42, reason=timeout, duration=5023ms

Grok pattern:

%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:thread}\] %{LOGLEVEL:level} %{NOTSPACE:logger} - %{GREEDYDATA:log_message}

Logstash config:

filter {
  grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:thread}\] %{LOGLEVEL:level} %{NOTSPACE:logger} - %{GREEDYDATA:log_message}"
    }
  }

  # Further parse the log message for key=value pairs
  kv {
    source => "log_message"
    field_split => ", "
    value_split => "="
    target => "parsed"
  }
}

Parsed output:

{
  "timestamp": "2026-04-01 10:15:30.456",
  "thread": "main",
  "level": "ERROR",
  "logger": "com.example.UserService",
  "log_message": "Failed to fetch user id=42, reason=timeout, duration=5023ms",
  "parsed": {
    "id": "42",
    "reason": "timeout",
    "duration": "5023ms"
  }
}

Grok Debugger

Kibana Grok Debugger

Kibana includes a built-in Grok Debugger under Dev Tools → Grok Debugger:

  1. Open Kibana → Management → Dev Tools
  2. Click Grok Debugger tab
  3. Paste your sample log message
  4. Enter your Grok pattern
  5. Click Simulate to see parsed output

Online Grok Debugger Tools

These tools let you test Grok patterns without a running Logstash or Elasticsearch instance. They are invaluable when building complex patterns.

Pattern Directory

Logstash loads patterns from the default patterns directory. You can add custom patterns using the patterns_dir option:

filter {
  grok {
    patterns_dir => ["/etc/logstash/patterns"]
    match => {
      "message" => "%{MY_CUSTOM_PATTERN:parsed}"
    }
  }
}

Pattern files are plain text, one pattern per line:

MY_CUSTOM_PATTERN %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}

Performance Tips

Tip Description
Anchor patterns Use ^ and $ to avoid backtracking
Avoid GREEDYDATA at the start Greedy patterns at the start cause excessive backtracking
Use COMBINEDAPACHELOG Pre-built patterns are optimized
Limit match patterns Fewer patterns = faster matching
Consider dissect first For delimiter-based logs, dissect is 5-10x faster
Use break_on_match Stop after first successful pattern match

Quick Reference

# Grok syntax
%{PATTERN:field_name}
%{PATTERN:field_name:data_type}          # int, float, boolean

# Common patterns
%{IP:client_ip}                          # IP address
%{WORD:method}                           # Single word
%{NUMBER:status:int}                     # Number → integer
%{TIMESTAMP_ISO8601:ts}                  # ISO timestamp
%{COMBINEDAPACHELOG}                     # Full Apache log
%{GREEDYDATA:msg}                        # Match everything

# Logstash grok filter
filter {
  grok {
    match => { "message" => "PATTERN" }
    break_on_match => true
    tag_on_failure => ["_grokparsefailure"]
    patterns_dir => ["/etc/logstash/patterns"]
  }
}

# ES ingest grok processor
{
  "grok": {
    "field": "message",
    "patterns": ["PATTERN"]
  }
}

Previous: Log Parsing ← · Next: Custom Grok Patterns →

You may also enjoy