ELK Stack — Elastic Observability

3 minute read

Elastic Observability

Elastic Observability unifies logs, metrics, traces, and uptime data into a single platform built on top of Elasticsearch and Kibana. Instead of switching between disconnected tools, you get correlated insights across all your telemetry data in one place.

The Three Pillars (Plus Uptime)

Traditional observability rests on three pillars. Elastic adds a fourth:

Pillar What It Captures Elastic Tool
Logs Application and system log events Filebeat, Elastic Agent
Metrics CPU, memory, disk, network, custom metrics Metricbeat, Elastic Agent
Traces Distributed request flows across services APM agents, OpenTelemetry
Uptime Endpoint availability and response times Heartbeat, Synthetics

Together, these pillars let you answer: What happened? Where did it happen? Why did it happen?

Unified Observability in Kibana

Kibana’s Observability section provides purpose-built UIs:

Logs Explorer

The Logs Explorer replaces the classic Logs UI with a powerful, fast interface for searching and filtering log data:

  • Stream logs in real time across all data sources
  • Filter by service, host, container, or custom fields
  • Correlate log entries with traces using trace.id
  • Use the integrated field statistics to spot anomalies

Infrastructure App

The Infrastructure app provides a real-time view of your hosts, containers, and pods:

  • Inventory view — See all hosts, containers, and pods at a glance with health indicators
  • Metrics Explorer — Build custom metric visualizations with grouping and filtering
  • Host/Container detail — Drill into individual resources to see processes, logs, and metrics

Service Map

The Service Map automatically discovers and visualizes relationships between your services:

  • View request flow between microservices
  • Identify latency bottlenecks and error hotspots
  • Click any service node to see transactions, errors, and dependencies

OpenTelemetry Integration

Elastic natively supports OpenTelemetry (OTel), the CNCF standard for telemetry:

# otel-collector-config.yaml — Export to Elasticsearch
exporters:
  elasticsearch:
    endpoints: ["https://your-elasticsearch:9200"]
    api_key: "your-api-key"
    logs_index: "otel-logs"
    traces_index: "otel-traces"

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elasticsearch]
    logs:
      receivers: [otlp]
      exporters: [elasticsearch]

Benefits of using OpenTelemetry with Elastic:

  • Vendor-neutral instrumentation — Switch backends without re-instrumenting
  • Wide language support — Java, Python, Go, .NET, JavaScript, Ruby, PHP, and more
  • Automatic instrumentation — Zero-code agents for many frameworks
  • Standards-based — OTLP protocol, semantic conventions, and baggage propagation

Elastic Synthetics

Elastic Synthetics lets you proactively monitor user journeys and API endpoints:

# Install the Synthetics CLI
npm install -g @elastic/synthetics

# Initialize a new project
npx @elastic/synthetics init my-synthetics-project
cd my-synthetics-project

Example synthetic monitor using Playwright:

// journeys/login-flow.journey.ts
import { journey, step, monitor } from '@elastic/synthetics';

journey('Login Flow', ({ page }) => {
  monitor.use({ schedule: '@every 5m' });

  step('Go to login page', async () => {
    await page.goto('https://myapp.example.com/login');
  });

  step('Enter credentials', async () => {
    await page.fill('#username', 'testuser');
    await page.fill('#password', 'testpass');
    await page.click('#login-button');
  });

  step('Verify dashboard loads', async () => {
    await page.waitForSelector('.dashboard-container');
  });
});

Synthetics monitors can:

  • Run from multiple global locations (Elastic Cloud) or private locations
  • Capture screenshots and network timings on failure
  • Feed data into SLO tracking and alerting

SLOs (Service Level Objectives)

Elastic lets you define and track SLOs directly in Kibana:

SLO Component Description Example
SLI Service Level Indicator (the metric) Percentage of requests < 500ms
Target Desired level of reliability 99.5%
Time Window Measurement period Rolling 30 days
Burn Rate How fast you’re consuming error budget 2× means budget consumed in 15 days

To create an SLO:

  1. Navigate to Observability → SLOs in Kibana
  2. Choose an SLI type (custom KQL, APM latency, or APM availability)
  3. Define target percentage and time window
  4. Configure burn-rate alerting rules

SLOs help bridge the gap between engineering metrics and business reliability goals.

Setting Up Elastic Observability

# Download and install Elastic Agent
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.17.0-linux-x86_64.tar.gz
cd elastic-agent-8.17.0-linux-x86_64

# Enroll with Fleet Server
sudo ./elastic-agent install \
  --url=https://fleet-server:8220 \
  --enrollment-token=YOUR_ENROLLMENT_TOKEN

Option 2: Individual Beats

# Install Filebeat for logs
sudo apt-get install filebeat
sudo filebeat modules enable system nginx

# Install Metricbeat for metrics
sudo apt-get install metricbeat
sudo metricbeat modules enable system docker kubernetes

# Install Heartbeat for uptime
sudo apt-get install heartbeat-elastic

Quick Reference

Elastic Observability at a Glance

Feature Purpose
Logs Explorer Search, filter, and stream log data
Infrastructure App Monitor hosts, containers, and pods
Service Map Visualize microservice dependencies
APM Application-level traces and transactions
Synthetics Proactive endpoint and journey monitoring
SLOs Define and track service reliability targets
Uptime Monitor endpoint availability via Heartbeat
OpenTelemetry Vendor-neutral telemetry ingestion
Alerting Rules-based notifications across all signals

Key Concept: Observability is not just monitoring. Monitoring tells you when something is wrong. Observability helps you understand why.

Summary

Elastic Observability brings logs, metrics, traces, and uptime together under a single platform. With native OpenTelemetry support, Synthetics monitoring, SLO tracking, and purpose-built Kibana UIs, you can move from reactive firefighting to proactive reliability engineering. In the next section, we’ll dive deeper into Elastic APM for application-level performance monitoring.

You may also enjoy