ELK Stack — Alerting

6 minute read

Introduction

Kibana Alerting allows you to define rules that periodically evaluate conditions against your Elasticsearch data and trigger actions when thresholds are breached. Whether you need to page an on-call engineer at 3 AM because error rates spiked, or send a weekly Slack summary of cluster health, Kibana Alerting is the engine behind it.

This section covers rule types, creating alert rules step-by-step, action connectors, alert management, maintenance windows, and rule snoozing.

Alerting Architecture

Kibana Alerting has three core components:

Component Description
Rule Defines what to check, how often, and when to alert
Connector Defines where to send notifications (Slack, email, PagerDuty, etc.)
Action Links a rule state (active, recovered) to a connector with a message template

The evaluation flow:

  1. Kibana’s Task Manager runs the rule on the configured schedule
  2. The rule queries Elasticsearch and evaluates conditions
  3. If conditions are met → alert status becomes Active
  4. Active alerts trigger configured actions via connectors
  5. When conditions clear → alert status becomes Recovered

Rule Types

Kibana provides several built-in rule types. The most commonly used ones:

Elasticsearch Query Rule

Fires when an Elasticsearch query returns results matching your conditions.

  • Use case: Alert when a specific error message appears in logs
  • Query types: KQL, Lucene, or Elasticsearch Query DSL
  • Conditions: Number of matching documents exceeds a threshold

Index Threshold Rule

Fires when an aggregated metric crosses a threshold.

  • Use case: Alert when average response time exceeds 2 seconds
  • Aggregations: Count, average, sum, min, max
  • Conditions: Is above, is below, is between, etc.

Log Threshold Rule (Observability)

Fires when the rate of log entries matching a condition exceeds a threshold.

  • Use case: Alert when more than 100 ERROR logs per minute
  • Requires: Observability plugin and log data

Metric Threshold Rule (Observability)

Fires when infrastructure metrics cross a threshold.

  • Use case: Alert when CPU usage > 90% for 5 minutes
  • Requires: Metrics data from Metricbeat or Elastic Agent

Anomaly Detection Rule (Machine Learning)

Fires when an ML anomaly score exceeds a severity threshold.

  • Use case: Alert on unusual login patterns detected by ML jobs

Creating an Alert Rule Step-by-Step

Here is a complete walkthrough of creating an Index Threshold rule that alerts when the error count exceeds 50 in a 5-minute window.

Step 1 — Navigate to Rules

  1. Open Stack Management → Rules (or Observability → Alerts → Manage Rules)
  2. Click Create rule

Step 2 — Configure Rule Details

  1. Name: High Error Rate — Production
  2. Tags: production, errors, p1
  3. Check every: 1 minute
  4. Notify: On status changes (avoids alert storms)

Step 3 — Select Rule Type

  1. Choose Index threshold
  2. Configure the condition:
    • Index: logs-*
    • Timestamp field: @timestamp
    • Aggregation: count()
    • When: is above
    • Threshold: 50
    • Over the last: 5 minutes
    • Group by: host.name (optional — creates per-host alerts)

Step 4 — Add Actions

  1. Click Add action
  2. Select a connector type (e.g., Slack)
  3. Choose an existing connector or create a new one
  4. Configure the action:
    • Run when: Alert is active
    • Message:
🚨 High Error Rate Alert

Host: {{context.group}}
Error count: {{context.value}} in last 5 minutes
Threshold: {{context.threshold}}
Timestamp: {{context.date}}

View in Kibana: {{context.link}}
  1. Optionally add a recovery action (when alert resolves):
    • Run when: Recovered
    • Message: ✅ Error rate for has returned to normal.

Step 5 — Save

Click Save to create the rule. It will begin evaluating immediately.

Action Connectors

Connectors are reusable integrations that actions use to deliver notifications.

Built-in Connector Types

Connector Description Common Use
Email SMTP email delivery Team distribution lists
Slack Post to Slack channel or DM Real-time team notifications
PagerDuty Create/resolve PD incidents On-call incident management
Webhook HTTP POST to any URL Custom integrations
Jira Create Jira issues Ticket-driven workflows
ServiceNow Create ServiceNow incidents Enterprise ITSM
Microsoft Teams Post to Teams channel Team notifications
Opsgenie Create Opsgenie alerts Incident management
Server log Write to Kibana server log Debugging and audit
Index Write alert data to an ES index Alert history and analytics

Creating a Slack Connector

  1. Open Stack Management → Connectors
  2. Click Create connector
  3. Select Slack
  4. Enter a name — e.g., SRE Slack Channel
  5. Choose API type:
    • Webhook URL — paste a Slack incoming webhook URL
    • Slack API — use a Slack Bot token for richer features
  6. Click Save & test
  7. Send a test message to verify

Creating a PagerDuty Connector

  1. Open Stack Management → Connectors
  2. Click Create connector → PagerDuty
  3. Enter the API URL — default: https://events.pagerduty.com
  4. Enter the Integration Key (from your PagerDuty service)
  5. Click Save & test

Creating a Webhook Connector

For custom integrations (e.g., posting to a Discord channel or internal API):

  1. Open Stack Management → Connectors
  2. Click Create connector → Webhook
  3. Configure:
    • URL: https://your-api.example.com/alerts
    • Method: POST
    • Headers: Content-Type: application/json
    • Authentication: Basic, SSL, or none
  4. Click Save & test

Action Frequency

Control how often actions fire to avoid alert fatigue:

Option Behavior
On status changes Fire once when alert activates, once when it recovers
On every check Fire every time the rule evaluates and the alert is active
Custom frequency Fire at most every N minutes/hours while alert is active

Best practice: Use On status changes for paging connectors (PagerDuty) and Custom frequency (every 1 hour) for email summaries.

Alert Status and Management

Alert States

State Meaning
Active Conditions are met; actions are firing
Recovered Conditions are no longer met; recovery actions fire
Untracked Alert was manually untracked; no actions fire
Flapping Alert is rapidly toggling between active and recovered

Managing Alerts

  1. Open Stack Management → Rules to see all rules
  2. Click a rule to view its alert history
  3. Use the Alert table to see individual alert instances
  4. Mute an individual alert to suppress actions without disabling the rule

Maintenance Windows

Maintenance windows allow you to suppress alert notifications during planned downtime.

Creating a Maintenance Window

  1. Open Stack Management → Maintenance Windows
  2. Click Create maintenance window
  3. Configure:
    • Name: Monthly Patch Window
    • Schedule: Set start and end time
    • Recurrence: One-time, daily, weekly, monthly, or custom
    • Filter: Optionally limit to specific rule tags (e.g., production)
  4. Click Save

During the maintenance window, rules continue to evaluate, but actions are suppressed. Alert state changes are still recorded.

Example — Recurring Weekly Maintenance

For a weekly deployment window every Thursday 2 AM–4 AM UTC:

  1. Name: Weekly Deployment Window
  2. Start: Next Thursday, 02:00 UTC
  3. End: Next Thursday, 04:00 UTC
  4. Recurrence: Weekly on Thursday
  5. Filter by tags: production

Rule Snoozing

Snoozing temporarily silences a specific rule without creating a full maintenance window.

How to Snooze a Rule

  1. Open Stack Management → Rules
  2. Find the rule you want to snooze
  3. Click the three-dot menu → Snooze
  4. Select duration:
    • 1 hour, 3 hours, 8 hours, 1 day
    • Custom end time
    • Indefinitely (until manually un-snoozed)
  5. The rule icon shows a snooze badge

Snoozing vs. Disabling vs. Muting

Action Rule evaluates? Actions fire? Use case
Enabled Yes Yes Normal operation
Snoozed Yes No Temporary suppression
Muted alert Yes No (for that alert) Suppress one alert instance
Disabled No No Full stop

Example Configurations

Production Error Rate Alert with Multi-Channel Notification

A platform team sets up tiered alerting:

Rule: Critical Error Rate

  • Check every: 1 minute
  • Index threshold on logs-*
  • Condition: count() > 100 over last 5 minutes, grouped by service.name

Actions:

  1. PagerDuty — runs on status change → creates/resolves incidents
  2. Slack #sre-alerts — runs on status change → posts alert details
  3. Index connector → writes alert data to alert-history-* for dashboarding

Maintenance window: Suppress during weekly Thursday 2–4 AM deployment window.

Daily Report via Email

Rule: Daily Log Summary

  • Check every: 24 hours (at 08:00 UTC)
  • ES query: Always matches (ensures daily execution)

Action:

  • Email connector → sends summary to team@company.com
  • Message template includes count of errors, warnings, and top error messages

Quick Reference

Task Steps
Create a rule Stack Management → Rules → Create rule
Create a connector Stack Management → Connectors → Create connector
Set action frequency Rule → Action → Run when → choose frequency
View alert history Stack Management → Rules → click rule → Alerts tab
Create maintenance window Stack Management → Maintenance Windows → Create
Snooze a rule Rules → three-dot menu → Snooze
Mute an individual alert Rule → Alerts tab → Mute
Test a connector Stack Management → Connectors → Test tab
Filter rules by tag Rules list → Tags filter

Next: Reporting and Spaces →

You may also enjoy