ELK Stack — Beats

2 minute read

Beats — Lightweight Data Shippers

Beats are a family of lightweight, single-purpose data shippers built by Elastic. They are installed as agents on your servers to collect and forward operational data — logs, metrics, network data, uptime information, and more — to Elasticsearch or Logstash.

Each Beat is designed to do one thing well. They use minimal system resources, making them ideal for deployment across hundreds or thousands of machines without impacting application performance.

┌─────────────────────────────────────────────────────┐
│                   Your Servers                       │
│                                                      │
│  ┌───────────┐  ┌────────────┐  ┌───────────────┐  │
│  │ Filebeat   │  │ Metricbeat │  │  Heartbeat    │  │
│  │ (logs)     │  │ (metrics)  │  │  (uptime)     │  │
│  └─────┬─────┘  └─────┬──────┘  └──────┬────────┘  │
│        │              │                 │            │
└────────┼──────────────┼─────────────────┼────────────┘
         │              │                 │
         ▼              ▼                 ▼
   ┌─────────────────────────────────────────────┐
   │           Logstash  or  Elasticsearch       │
   └─────────────────────────────────────────────┘
                        │
                        ▼
               ┌─────────────────┐
               │     Kibana      │
               └─────────────────┘

What You’ll Learn

In this chapter you will learn how to:

  • Install and configure Filebeat to collect and ship log files from any source
  • Use Filebeat modules to quickly set up log collection for common services (Nginx, Apache, MySQL, system logs)
  • Deploy Metricbeat to collect system and service metrics (CPU, memory, disk, containers)
  • Set up Heartbeat to monitor service uptime via HTTP, TCP, and ICMP checks
  • Use processors to enrich, filter, and transform data before it leaves the agent
  • Understand Elastic Agent and Fleet — the modern unified approach to data collection

Chapter Contents

# Topic Description
1 Filebeat Architecture, configuration, inputs, outputs, multiline handling
2 Filebeat Modules Pre-built modules for common log formats and services
3 Metricbeat and Heartbeat System metrics collection and uptime monitoring
4 Processors Data enrichment, filtering, and transformation at the edge
5 Elastic Agent and Fleet Unified agent management and centralized Fleet control
6 Practice Exercises Hands-on exercises to reinforce Beats concepts

Why Beats?

Characteristic Detail
Lightweight Written in Go — small memory footprint (~20-50 MB)
Reliable At-least-once delivery with registry-based tracking
Extensible Processors allow edge-side enrichment and filtering
Modular Pre-built modules for 50+ common data sources
Secure TLS/SSL, API key, and certificate-based authentication
Scalable Deploy across thousands of hosts with minimal overhead

Prerequisites

Before starting this chapter, make sure you have:

  • A running Elasticsearch cluster (see Chapter 2)
  • Kibana installed and accessible (see Chapter 4)
  • Basic understanding of YAML configuration syntax
  • Root or sudo access on the server where you will install Beats

Let’s begin with Filebeat — the most widely used Beat for log collection.

You may also enjoy