ELK Stack — Analyzers and Tokenizers

Analyzers and Tokenizers

When you index a text field, Elasticsearch does not store the raw string as-is. It runs the value through an analysis pipeline that breaks the text into individual tokens (terms) and applies transformations. Understanding this pipeline is the key to getting relevant search results.

---

The Analysis Pipeline

Every analyzer is composed of three stages that execute in order:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   CHARACTER      │     │                 │     │   TOKEN          │
│   FILTERS        │────▶│   TOKENIZER     │────▶│   FILTERS        │
│                 │     │                 │     │                 │
│ • html_strip    │     │ • standard      │     │ • lowercase     │
│ • mapping       │     │ • keyword       │     │ • stop          │
│ • pattern_replace│    │ • pattern       │     │ • stemmer       │
│                 │     │ • edge_ngram    │     │ • synonym       │
└─────────────────┘     └─────────────────┘     └─────────────────┘

Input: "<p>The Quick FOXES jumped!</p>"
  ↓ char filter (html_strip)
       "The Quick FOXES jumped!"
  ↓ tokenizer (standard)
       ["The", "Quick", "FOXES", "jumped"]
  ↓ token filters (lowercase, stop)
       ["quick", "foxes", "jumped"]

• Character filters modify the raw text before tokenization (e.g., strip HTML).
• Tokenizer splits the text into tokens.
• Token filters transform each token (e.g., lowercase, remove stop words, stem).

---

Testing with the _analyze API

The _analyze API lets you test any analyzer without creating an index:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "standard",
  "text": "The Quick Brown Foxes jumped over the lazy dog!"
}'

{
  "tokens": [
    { "token": "the",    "start_offset": 0,  "end_offset": 3,  "type": "<ALPHANUM>", "position": 0 },
    { "token": "quick",  "start_offset": 4,  "end_offset": 9,  "type": "<ALPHANUM>", "position": 1 },
    { "token": "brown",  "start_offset": 10, "end_offset": 15, "type": "<ALPHANUM>", "position": 2 },
    { "token": "foxes",  "start_offset": 16, "end_offset": 21, "type": "<ALPHANUM>", "position": 3 },
    { "token": "jumped", "start_offset": 22, "end_offset": 28, "type": "<ALPHANUM>", "position": 4 },
    { "token": "over",   "start_offset": 29, "end_offset": 33, "type": "<ALPHANUM>", "position": 5 },
    { "token": "the",    "start_offset": 34, "end_offset": 37, "type": "<ALPHANUM>", "position": 6 },
    { "token": "lazy",   "start_offset": 38, "end_offset": 42, "type": "<ALPHANUM>", "position": 7 },
    { "token": "dog",    "start_offset": 43, "end_offset": 46, "type": "<ALPHANUM>", "position": 8 }
  ]
}

The standard analyzer lowercases and splits on word boundaries but does not remove stop words.

---

Built-in Analyzers

Standard Analyzer

The default. Splits on word boundaries, lowercases, and optionally removes stop words.

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "standard",
  "text": "user.email@example.com"
}'

{
  "tokens": [
    { "token": "user.email", "start_offset": 0,  "end_offset": 10, "type": "<ALPHANUM>", "position": 0 },
    { "token": "example.com", "start_offset": 11, "end_offset": 22, "type": "<ALPHANUM>", "position": 1 }
  ]
}

Simple Analyzer

Splits on non-letter characters and lowercases.

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "simple",
  "text": "Error-404: Page not found!"
}'

{
  "tokens": [
    { "token": "error",   "start_offset": 0,  "end_offset": 5,  "type": "word", "position": 0 },
    { "token": "page",    "start_offset": 11, "end_offset": 15, "type": "word", "position": 1 },
    { "token": "not",     "start_offset": 16, "end_offset": 19, "type": "word", "position": 2 },
    { "token": "found",   "start_offset": 20, "end_offset": 25, "type": "word", "position": 3 }
  ]
}

Notice that 404 is dropped because the simple analyzer only keeps letters.

Whitespace Analyzer

Splits on whitespace only — no lowercasing, no punctuation handling.

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "whitespace",
  "text": "GET /api/users HTTP/1.1"
}'

{
  "tokens": [
    { "token": "GET",       "start_offset": 0,  "end_offset": 3,  "type": "word", "position": 0 },
    { "token": "/api/users", "start_offset": 4, "end_offset": 14, "type": "word", "position": 1 },
    { "token": "HTTP/1.1",  "start_offset": 15, "end_offset": 23, "type": "word", "position": 2 }
  ]
}

Keyword Analyzer

Does nothing — the entire input becomes a single token. Useful when you want exact matching on a text field.

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "keyword",
  "text": "Order-12345-US"
}'

{
  "tokens": [
    { "token": "Order-12345-US", "start_offset": 0, "end_offset": 14, "type": "word", "position": 0 }
  ]
}

Pattern Analyzer

Splits using a regex pattern (default: \W+ — non-word characters).

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "pattern",
  "text": "foo-bar_baz.qux"
}'

{
  "tokens": [
    { "token": "foo", "start_offset": 0,  "end_offset": 3,  "type": "word", "position": 0 },
    { "token": "bar", "start_offset": 4,  "end_offset": 7,  "type": "word", "position": 1 },
    { "token": "baz", "start_offset": 8,  "end_offset": 11, "type": "word", "position": 2 },
    { "token": "qux", "start_offset": 12, "end_offset": 15, "type": "word", "position": 3 }
  ]
}

Language Analyzers

Elasticsearch includes analyzers for many languages. The english analyzer applies stemming and stop word removal:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "english",
  "text": "The foxes were running quickly through the forests"
}'

{
  "tokens": [
    { "token": "fox",    "start_offset": 4,  "end_offset": 9,  "type": "<ALPHANUM>", "position": 1 },
    { "token": "were",   "start_offset": 10, "end_offset": 14, "type": "<ALPHANUM>", "position": 2 },
    { "token": "run",    "start_offset": 15, "end_offset": 22, "type": "<ALPHANUM>", "position": 3 },
    { "token": "quickli","start_offset": 23, "end_offset": 30, "type": "<ALPHANUM>", "position": 4 },
    { "token": "through","start_offset": 31, "end_offset": 38, "type": "<ALPHANUM>", "position": 5 },
    { "token": "forest", "start_offset": 43, "end_offset": 50, "type": "<ALPHANUM>", "position": 7 }
  ]
}

Stop words (the) are removed. foxes → fox, forests → forest (stemmed).

---

Character Filters

html_strip

Removes HTML tags:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "char_filter": ["html_strip"],
  "tokenizer": "standard",
  "text": "<p>Hello <b>World</b></p>"
}'

{
  "tokens": [
    { "token": "hello", "start_offset": 3,  "end_offset": 12, "type": "<ALPHANUM>", "position": 0 },
    { "token": "world", "start_offset": 13, "end_offset": 25, "type": "<ALPHANUM>", "position": 1 }
  ]
}

mapping

Replaces characters based on a mapping:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "char_filter": [
    {
      "type": "mapping",
      "mappings": [":)" => " happy ", ":(" => " sad "]
    }
  ],
  "tokenizer": "standard",
  "text": "I am feeling :) today"
}'

pattern_replace

Uses regex to replace characters:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "char_filter": [
    {
      "type": "pattern_replace",
      "pattern": "[^a-zA-Z0-9 ]",
      "replacement": ""
    }
  ],
  "tokenizer": "standard",
  "text": "order#12345-US!"
}'

{
  "tokens": [
    { "token": "order12345us", "start_offset": 0, "end_offset": 15, "type": "<ALPHANUM>", "position": 0 }
  ]
}

---

Tokenizers

edge_ngram

Creates tokens from the beginning of the input — ideal for autocomplete:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "tokenizer": {
    "type": "edge_ngram",
    "min_gram": 2,
    "max_gram": 5
  },
  "text": "search"
}'

{
  "tokens": [
    { "token": "se",    "start_offset": 0, "end_offset": 6, "type": "word", "position": 0 },
    { "token": "sea",   "start_offset": 0, "end_offset": 6, "type": "word", "position": 1 },
    { "token": "sear",  "start_offset": 0, "end_offset": 6, "type": "word", "position": 2 },
    { "token": "searc", "start_offset": 0, "end_offset": 6, "type": "word", "position": 3 }
  ]
}

ngram

Similar to edge_ngram but generates grams from every position:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "tokenizer": {
    "type": "ngram",
    "min_gram": 3,
    "max_gram": 3
  },
  "text": "hello"
}'

{
  "tokens": [
    { "token": "hel", "start_offset": 0, "end_offset": 5, "type": "word", "position": 0 },
    { "token": "ell", "start_offset": 0, "end_offset": 5, "type": "word", "position": 1 },
    { "token": "llo", "start_offset": 0, "end_offset": 5, "type": "word", "position": 2 }
  ]
}

---

Token Filters

lowercase

Converts all tokens to lowercase (included by default in the standard analyzer).

stop

Removes common stop words:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "tokenizer": "standard",
  "filter": ["lowercase", "stop"],
  "text": "The quick brown fox is a mammal"
}'

{
  "tokens": [
    { "token": "quick",  "start_offset": 4,  "end_offset": 9,  "type": "<ALPHANUM>", "position": 1 },
    { "token": "brown",  "start_offset": 10, "end_offset": 15, "type": "<ALPHANUM>", "position": 2 },
    { "token": "fox",    "start_offset": 16, "end_offset": 19, "type": "<ALPHANUM>", "position": 3 },
    { "token": "mammal", "start_offset": 25, "end_offset": 31, "type": "<ALPHANUM>", "position": 6 }
  ]
}

synonym

Maps tokens to synonyms:

curl -s -X PUT "localhost:9200/products" \
  -H "Content-Type: application/json" \
  -d '{
  "settings": {
    "analysis": {
      "filter": {
        "my_synonyms": {
          "type": "synonym",
          "synonyms": [
            "laptop, notebook, portable computer",
            "phone, mobile, cellphone"
          ]
        }
      },
      "analyzer": {
        "synonym_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "my_synonyms"]
        }
      }
    }
  }
}'

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "products"
}

stemmer

Reduces words to their root form:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "tokenizer": "standard",
  "filter": ["lowercase", { "type": "stemmer", "language": "english" }],
  "text": "running runners ran"
}'

{
  "tokens": [
    { "token": "run",  "start_offset": 0,  "end_offset": 7,  "type": "<ALPHANUM>", "position": 0 },
    { "token": "runner","start_offset": 8,  "end_offset": 15, "type": "<ALPHANUM>", "position": 1 },
    { "token": "ran",  "start_offset": 16, "end_offset": 19, "type": "<ALPHANUM>", "position": 2 }
  ]
}

asciifolding

Converts Unicode characters to ASCII equivalents:

curl -s -X POST "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "tokenizer": "standard",
  "filter": ["asciifolding"],
  "text": "café résumé naïve"
}'

{
  "tokens": [
    { "token": "cafe",   "start_offset": 0,  "end_offset": 4,  "type": "<ALPHANUM>", "position": 0 },
    { "token": "resume", "start_offset": 5,  "end_offset": 11, "type": "<ALPHANUM>", "position": 1 },
    { "token": "naive",  "start_offset": 12, "end_offset": 17, "type": "<ALPHANUM>", "position": 2 }
  ]
}

---

Building a Custom Analyzer

Create an index with a custom analyzer that strips HTML, tokenizes with standard tokenizer, lowercases, removes English stop words, and applies stemming:

curl -s -X PUT "localhost:9200/articles" \
  -H "Content-Type: application/json" \
  -d '{
  "settings": {
    "analysis": {
      "char_filter": {
        "strip_html": {
          "type": "html_strip"
        }
      },
      "analyzer": {
        "article_analyzer": {
          "type": "custom",
          "char_filter": ["strip_html"],
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stop", "english_stemmer"]
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        },
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title":   { "type": "text", "analyzer": "article_analyzer" },
      "body":    { "type": "text", "analyzer": "article_analyzer" },
      "author":  { "type": "keyword" }
    }
  }
}'

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "articles"
}

Test the Custom Analyzer

curl -s -X POST "localhost:9200/articles/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
  "analyzer": "article_analyzer",
  "text": "<p>The developers are <b>building</b> distributed systems</p>"
}'

{
  "tokens": [
    { "token": "develop",     "start_offset": 3,  "end_offset": 17, "type": "<ALPHANUM>", "position": 1 },
    { "token": "build",       "start_offset": 22, "end_offset": 40, "type": "<ALPHANUM>", "position": 3 },
    { "token": "distribut",   "start_offset": 41, "end_offset": 52, "type": "<ALPHANUM>", "position": 4 },
    { "token": "system",      "start_offset": 53, "end_offset": 63, "type": "<ALPHANUM>", "position": 5 }
  ]
}

HTML was stripped, stop words (the, are) were removed, and words were stemmed (developers → develop, building → build).

---

Autocomplete Analyzer Example

A common pattern for search-as-you-type uses edge_ngram at index time and standard at search time:

curl -s -X PUT "localhost:9200/autocomplete-demo" \
  -H "Content-Type: application/json" \
  -d '{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_index": {
          "type": "custom",
          "tokenizer": "autocomplete_tokenizer",
          "filter": ["lowercase"]
        },
        "autocomplete_search": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 15,
          "token_chars": ["letter", "digit"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "suggest": {
        "type": "text",
        "analyzer": "autocomplete_index",
        "search_analyzer": "autocomplete_search"
      }
    }
  }
}'

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "autocomplete-demo"
}

---

Quick Reference

Analyzer	Tokenizer	Token Filters	Use Case
standard	Standard (Unicode)	Lowercase	General-purpose text
simple	Letter tokenizer	Lowercase	Text without numbers
whitespace	Whitespace	None	Structured fields, log lines
keyword	Keyword (noop)	None	Exact matching on text fields
pattern	Regex pattern	Lowercase	Custom delimiters
english	Standard	Lowercase, stop, stemmer	English-language content

Component	Type	Examples
Character filters	html_strip, mapping, pattern_replace	Strip tags, replace emojis
Tokenizers	standard, keyword, pattern, edge_ngram, ngram	Word boundaries, autocomplete
Token filters	lowercase, stop, stemmer, synonym, asciifolding	Normalize, remove, map

---

Next → Ingest Pipelines