ELK Stack — Elastic Agent and Fleet

6 minute read

Elastic Agent Overview

Elastic Agent is a single, unified agent that replaces the need to install and manage multiple individual Beats on each host. Introduced in Elastic 7.x and fully mature in 8.x, it provides a single binary that can collect logs, metrics, security data, and uptime information — all managed from a central point.

Note: Individual Beats (Filebeat, Metricbeat, Heartbeat, etc.) are still fully supported. Elastic Agent is the modern recommended approach for new deployments starting with Elastic 8.x.

Why Elastic Agent?

With individual Beats, managing 5 different data shippers across hundreds of servers becomes complex. You need to install, configure, and update each Beat separately on every host. Elastic Agent solves this by consolidating everything into one agent with centralized management.

Fleet Architecture

Fleet is the centralized management UI in Kibana that lets you manage Elastic Agent deployments at scale. The architecture consists of:

┌──────────────────────────────────────────────────────────────┐
│                         Kibana                                │
│                                                               │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │                    Fleet UI                              │ │
│  │                                                          │ │
│  │  ┌──────────────┐  ┌──────────────┐  ┌───────────────┐ │ │
│  │  │   Policies    │  │ Integrations │  │   Agents      │ │ │
│  │  │              │  │              │  │   Status      │ │ │
│  │  │ - Web Servers │  │ - System     │  │ - 500 online │ │ │
│  │  │ - DB Servers  │  │ - Nginx      │  │ - 3 offline  │ │ │
│  │  │ - K8s Nodes   │  │ - Docker     │  │ - 2 updating │ │ │
│  │  └──────────────┘  │ - MySQL      │  └───────────────┘ │ │
│  │                     │ - K8s        │                     │ │
│  │                     └──────────────┘                     │ │
│  └─────────────────────┬───────────────────────────────────┘ │
│                        │                                      │
└────────────────────────┼──────────────────────────────────────┘
                         │ Agent Policies
                         ▼
              ┌──────────────────────┐
              │    Fleet Server      │
              │  (coordination hub)  │
              └──────────┬───────────┘
                         │
          ┌──────────────┼──────────────┐
          │              │              │
          ▼              ▼              ▼
   ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
   │ Elastic     │ │ Elastic     │ │ Elastic     │
   │ Agent       │ │ Agent       │ │ Agent       │
   │ (web-01)    │ │ (db-01)     │ │ (k8s-node1) │
   │             │ │             │ │             │
   │ ● Logs      │ │ ● Logs      │ │ ● Logs      │
   │ ● Metrics   │ │ ● Metrics   │ │ ● Metrics   │
   │ ● Uptime    │ │ ● MySQL     │ │ ● K8s       │
   │ ● Nginx     │ │ ● Uptime    │ │ ● Docker    │
   └─────────────┘ └─────────────┘ └─────────────┘

Key Concepts

Concept	Description
Fleet Server	A special instance of Elastic Agent that serves as the coordination hub between Kibana and all enrolled agents
Agent Policy	A configuration template that defines which integrations (data sources) an agent should collect
Integration	A pre-built package for a specific data source (System, Nginx, Docker, MySQL, etc.) — equivalent to a Beat module
Enrollment Token	A one-time token used to register a new Elastic Agent with Fleet Server

Individual Beats vs Elastic Agent

Feature	Individual Beats	Elastic Agent
Installation	Install each Beat separately (Filebeat, Metricbeat, etc.)	Single binary for all data collection
Configuration	Edit YAML files on each host	Centralized via Fleet UI in Kibana
Updates	Manually update each Beat on every host	Rolling upgrades from Fleet UI
Management	Decentralized — SSH into each host	Centralized — single pane of glass in Kibana
Monitoring	Monitor each Beat separately	Unified agent health dashboard
Data sources	One Beat per data type	Multiple integrations in one agent
Flexibility	Full control over YAML config	Policy-based, less low-level control
Maturity	Battle-tested, 10+ years	Mature in 8.x, actively developed
Standalone mode	Always standalone	Supports both standalone and Fleet-managed
Resource usage	~20-50 MB per Beat	~100-200 MB for one agent (replaces multiple Beats)
Best for	Simple setups, edge cases, full control needed	Large-scale deployments, centralized management

Installing Fleet Server

Fleet Server is the first Elastic Agent you install. It acts as the bridge between Kibana and all other agents.

# 1. Download Elastic Agent
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.17.0-linux-x86_64.tar.gz
cd elastic-agent-8.17.0-linux-x86_64

# 2. Install as Fleet Server
sudo ./elastic-agent install \
  --fleet-server-es=https://es-node-1:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aW... \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-port=8220 \
  --certificate-authorities=/etc/fleet/certs/ca.crt \
  --fleet-server-cert=/etc/fleet/certs/fleet-server.crt \
  --fleet-server-cert-key=/etc/fleet/certs/fleet-server.key

Generate the service token in Kibana: Management → Fleet → Settings → Fleet Server hosts → Generate service token.

Installing and Enrolling Elastic Agent

Once Fleet Server is running, enroll additional agents:

# 1. Download Elastic Agent on the target host
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.17.0-linux-x86_64.tar.gz
cd elastic-agent-8.17.0-linux-x86_64

# 2. Enroll with Fleet Server
sudo ./elastic-agent install \
  --url=https://fleet-server:8220 \
  --enrollment-token=RXhNbVpZOEJfVXFhLTdSX186... \
  --certificate-authorities=/etc/elastic-agent/certs/ca.crt

Get the enrollment token from Kibana: Fleet → Agent policies → [Your Policy] → Add agent → Enroll in Fleet.

After enrollment, the agent appears in Fleet and starts collecting data based on its assigned policy.

Agent Policies and Integrations

Creating a Policy in Kibana

Navigate to Management → Fleet → Agent policies
Click Create agent policy
Name it (e.g., “Web Servers Policy”)
Add integrations:
- System — collects system logs and metrics
- Nginx — collects Nginx access and error logs
- Docker — collects Docker container metrics

Policy Configuration Example

Each integration in a policy is configured through the Fleet UI. Behind the scenes, the configuration looks like:

# This is managed by Fleet — you don't edit this directly
# Shown here for understanding

inputs:
  - id: system-logs
    type: logfile
    data_stream:
      namespace: default
    streams:
      - id: system-syslog
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
          - /var/log/syslog
          - /var/log/messages

  - id: system-metrics
    type: system/metrics
    data_stream:
      namespace: default
    streams:
      - id: system-cpu
        data_stream:
          dataset: system.cpu
          type: metrics
        metricsets: ["cpu"]
        period: 10s

  - id: nginx-logs
    type: logfile
    data_stream:
      namespace: default
    streams:
      - id: nginx-access
        data_stream:
          dataset: nginx.access
          type: logs
        paths:
          - /var/log/nginx/access.log*

Standalone vs Fleet-Managed Mode

Elastic Agent can run in two modes:

Fleet-Managed (Recommended)

Configuration is pushed from Fleet Server
Centralized management in Kibana
Automatic updates and policy changes
Best for large-scale deployments

Standalone Mode

Configuration is local — you edit elastic-agent.yml on each host
No Fleet Server required
Full control over the configuration
Useful for air-gapped environments or when you need fine-grained control

# Install in standalone mode (no Fleet enrollment)
sudo ./elastic-agent install

# Edit the standalone config
sudo vi /etc/elastic-agent/elastic-agent.yml

Standalone elastic-agent.yml example:

outputs:
  default:
    type: elasticsearch
    hosts:
      - "https://es-node-1:9200"
    username: "elastic"
    password: "changeme"
    ssl:
      certificate_authorities:
        - "/etc/elastic-agent/certs/ca.crt"

inputs:
  - type: logfile
    id: system-logs
    data_stream:
      namespace: default
    streams:
      - id: syslog
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
          - /var/log/syslog

  - type: system/metrics
    id: system-metrics
    data_stream:
      namespace: default
    streams:
      - id: cpu
        data_stream:
          dataset: system.cpu
          type: metrics
        metricsets: ["cpu"]
        period: 10s

Managing Agents in Fleet

Common operations in the Fleet UI:

# Check agent status from the command line
sudo elastic-agent status

# View agent logs
sudo journalctl -u elastic-agent -f

# Restart the agent
sudo systemctl restart elastic-agent

# Unenroll an agent (from Kibana Fleet UI or CLI)
sudo elastic-agent uninstall

From Kibana Fleet:

View all agents: Fleet → Agents — see health status, version, policy, last checkin
Upgrade agents: Select agents → Actions → Upgrade — rolling upgrade without SSH
Reassign policy: Select agent → Actions → Assign to new policy
Unenroll: Select agent → Actions → Unenroll

Quick Reference

Task	Command / Location
Download agent	`elastic.co/downloads/elastic-agent`
Install Fleet Server	`elastic-agent install --fleet-server-es=...`
Enroll agent	`elastic-agent install --url=... --enrollment-token=...`
Check status	`elastic-agent status`
View logs	`journalctl -u elastic-agent -f`
Fleet UI	Kibana → Management → Fleet
Create policy	Fleet → Agent policies → Create
Add integration	Policy → Add integration
Generate token	Fleet → Enrollment tokens → Create
Upgrade agents	Fleet → Agents → Select → Upgrade
Standalone config	`/etc/elastic-agent/elastic-agent.yml`

Next: Practice Exercises — hands-on exercises to reinforce Beats concepts.

Share on

Twitter Facebook LinkedIn