ELK Stack — Processors

What Are Processors?

Processors are lightweight transformation plugins built into every Beat. They allow you to filter, enrich, transform, and drop events before they are sent to the output. Processors run on the Beat itself (at the edge), reducing the load on Logstash or Elasticsearch ingest pipelines.

Processors are defined in the processors section of the Beat configuration file and are executed in the order they are listed.

Why Use Processors?

Benefit	Description
Reduce data volume	Drop noisy or irrelevant events before shipping
Enrich at the edge	Add host, cloud, container, or Kubernetes metadata without Logstash
Improve security	Remove sensitive fields (passwords, tokens) before data leaves the server
Simplify pipelines	Handle common transformations without needing Logstash
Lower costs	Less data shipped = lower storage and network costs

Processor Ordering

Processors execute sequentially in the order they are defined. The output of one processor becomes the input of the next.

processors:
  # Step 1: Add hostname and OS info
  - add_host_metadata: ~

  # Step 2: Add cloud provider metadata (AWS, GCP, Azure)
  - add_cloud_metadata: ~

  # Step 3: Remove sensitive fields
  - drop_fields:
      fields: ["agent.ephemeral_id", "host.mac"]

  # Step 4: Drop empty or noisy events
  - drop_event:
      when:
        regexp:
          message: "^\\s*$"

add_fields

Add custom static fields to every event:

processors:
  - add_fields:
      target: ""
      fields:
        environment: production
        team: platform-engineering
        datacenter: us-east-1

  # Nested target — adds fields under a specific key
  - add_fields:
      target: project
      fields:
        name: web-api
        version: "2.5.0"
        owner: backend-team

Result: each event will contain environment: production, team: platform-engineering, and a nested project.name: web-api.

drop_event

Drop events entirely based on conditions:

processors:
  # Drop DEBUG-level log lines
  - drop_event:
      when:
        regexp:
          message: "^DEBUG"

  # Drop events from a specific container
  - drop_event:
      when:
        equals:
          container.name: "logspam-sidecar"

  # Drop health check requests from Nginx logs
  - drop_event:
      when:
        contains:
          message: "GET /healthz"

  # Drop events where HTTP status is 2xx (keep only errors)
  - drop_event:
      when:
        range:
          http.response.status_code:
            gte: 200
            lt: 300

drop_fields

Remove specific fields from events:

processors:
  - drop_fields:
      fields:
        - host.mac
        - host.ip
        - agent.ephemeral_id
        - ecs.version
      # ignore_missing: don't error if a field doesn't exist
      ignore_missing: true

rename

Rename fields in the event:

processors:
  - rename:
      fields:
        - from: "log.file.path"
          to: "source_file"
        - from: "host.hostname"
          to: "server_name"
      ignore_missing: true
      fail_on_error: false

include_fields

Keep only the specified fields and remove everything else. Useful for reducing event size:

processors:
  - include_fields:
      fields:
        - "@timestamp"
        - message
        - host.name
        - log.level
        - service.name
        - trace.id

add_host_metadata

Enrich every event with information about the host where the Beat is running:

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
      # Optionally specify which network interface to report
      # netinfo.enabled: true
      # cache.ttl: 5m
      # geo:
      #   name: us-east-dc
      #   location: "40.7128, -74.0060"

Adds fields like host.name, host.hostname, host.architecture, host.os.platform, host.os.version, host.ip, host.mac.

add_cloud_metadata

Automatically detect and add cloud provider metadata (works on AWS, GCP, Azure, and more):

processors:
  - add_cloud_metadata: ~

When running on AWS EC2, this adds:

• cloud.provider: aws
• cloud.instance.id: i-0abcd1234
• cloud.region: us-east-1
• cloud.availability_zone: us-east-1a
• cloud.machine.type: m5.xlarge
• cloud.account.id: 123456789012

add_docker_metadata

Add Docker container metadata to events:

processors:
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"
      # Match container by log file path (for Filebeat)
      match_source: true
      match_source_index: 4
      # Only add specific fields
      # match_fields: ["system.process.cgroup.id"]
      # match_pids: ["process.pid", "process.ppid"]

Adds fields like container.id, container.name, container.image.name, container.labels.*.

add_kubernetes_metadata

Add Kubernetes pod and container metadata:

processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      # Scope: node or cluster
      scope: node
      matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
      default_indexers.enabled: true
      default_matchers.enabled: true

Adds fields like kubernetes.pod.name, kubernetes.namespace, kubernetes.node.name, kubernetes.labels.*, kubernetes.container.name.

decode_json_fields

Parse JSON strings embedded inside a field:

processors:
  - decode_json_fields:
      fields: ["message"]
      # Merge decoded JSON into the root of the event
      target: ""
      # Overwrite existing keys if they conflict
      overwrite_keys: true
      # Expand dots in JSON keys into nested objects
      expand_keys: true
      # Add error if JSON parsing fails
      add_error_key: true
      # Max depth for nested JSON
      max_depth: 5

This is extremely useful when your application writes structured JSON logs. For example, if the message field contains:

{"level":"error","service":"payment-api","trace_id":"abc123","msg":"timeout"}

After processing, the event will have top-level fields: level, service, trace_id, msg.

dissect

Tokenize a string field using a dissect pattern (faster than regex):

processors:
  - dissect:
      tokenizer: '%{client_ip} %{ident} %{user} [%{timestamp}] "%{method} %{uri} %{protocol}" %{status} %{bytes}'
      field: "message"
      target_prefix: "apache"

Input: 192.168.1.50 - admin [01/Apr/2026:14:47:00 +0530] "GET /api/users HTTP/1.1" 200 1234

Result: apache.client_ip: 192.168.1.50, apache.method: GET, apache.uri: /api/users, apache.status: 200, etc.

Conditional Processing (when)

Most processors support a when condition to run only on matching events:

processors:
  # Only add Kubernetes metadata if running in K8s
  - add_kubernetes_metadata:
      when:
        has_fields: ["kubernetes"]

  # Drop events only if log level is DEBUG
  - drop_event:
      when:
        equals:
          log.level: "debug"

  # Add a field only for specific hosts
  - add_fields:
      when:
        regexp:
          host.name: "^prod-web-.*"
      target: ""
      fields:
        tier: frontend

  # Complex conditions with AND/OR/NOT
  - drop_event:
      when:
        and:
          - equals:
              http.response.status_code: 200
          - contains:
              url.path: "/healthz"
          - not:
              has_fields: ["error"]

Condition Types

Condition	Description	Example
equals	Exact match	equals: { log.level: "debug" }
contains	Substring match	contains: { message: "timeout" }
regexp	Regex match	regexp: { host.name: "^prod-.*" }
range	Numeric range	range: { http.response.status_code: { gte: 400 } }
has_fields	Field exists	has_fields: ["error.message"]
or	Logical OR	Combine multiple conditions
and	Logical AND	Combine multiple conditions
not	Logical NOT	Negate a condition

Complete Processor Chain Example

A real-world processor chain for a production Filebeat deployment:

processors:
  # 1. Parse JSON log lines
  - decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true
      add_error_key: true

  # 2. Add infrastructure metadata
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

  # 3. Add custom fields
  - add_fields:
      target: ""
      fields:
        environment: production
        cluster: us-east-primary

  # 4. Remove sensitive fields
  - drop_fields:
      fields:
        - "password"
        - "authorization"
        - "cookie"
        - "agent.ephemeral_id"
      ignore_missing: true

  # 5. Rename fields for consistency
  - rename:
      fields:
        - from: "level"
          to: "log.level"
        - from: "svc"
          to: "service.name"
      ignore_missing: true

  # 6. Drop noisy events
  - drop_event:
      when:
        or:
          - contains:
              message: "GET /healthz"
          - regexp:
              message: "^\\s*$"
          - and:
              - equals:
                  log.level: "debug"
              - not:
                  regexp:
                    service.name: "^payment-.*"

Quick Reference

Processor	Purpose
add_fields	Add static fields to events
drop_event	Discard events matching a condition
drop_fields	Remove specific fields from events
include_fields	Keep only listed fields, drop everything else
rename	Rename fields
add_host_metadata	Add hostname, OS, IP information
add_cloud_metadata	Add AWS/GCP/Azure cloud metadata
add_docker_metadata	Add Docker container info
add_kubernetes_metadata	Add K8s pod, namespace, labels
decode_json_fields	Parse JSON strings into structured fields
dissect	Tokenize strings using dissect patterns
Conditions:	equals, contains, regexp, range, has_fields, and, or, not

---

Next: Elastic Agent and Fleet — the modern unified approach to data collection.