Docker Compose — ELK Stack Project

Build a complete Elasticsearch, Logstash, and Kibana (ELK) stack for centralized logging — a must-know stack for any DevOps practitioner.

Architecture

┌────────────────────────────────────────────────────────┐
│                Docker Network: elk-net                   │
│                                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────┐  │
│  │ Elasticsearch │  │   Logstash   │  │    Kibana     │  │
│  │    :9200      │←─│    :5044     │  │    :5601      │  │
│  │    :9300      │  │    :9600     │  │               │  │
│  └──────────────┘  └──────────────┘  └───────────────┘  │
│        │                  │                              │
│    es-data           pipeline.conf                       │
│   (named vol)       (bind mount)                         │
└────────────────────────────────────────────────────────┘

Project Setup

mkdir elk-compose && cd elk-compose
mkdir logstash

Step 1 — Logstash Pipeline Configuration

Create logstash/pipeline.conf:

input {
  beats {
    port => 5044
  }
  tcp {
    port => 5000
    codec => json
  }
}

filter {
  if [message] =~ /^\{/ {
    json {
      source => "message"
    }
  }
  mutate {
    add_field => { "received_at" => "%{@timestamp}" }
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
  stdout {
    codec => rubydebug
  }
}

Step 2 — The Compose File

Create compose.yml:

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - elk-net
    deploy:
      resources:
        limits:
          memory: 1g
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 10
      start_period: 30s

  logstash:
    image: docker.elastic.co/logstash/logstash:8.12.0
    depends_on:
      elasticsearch:
        condition: service_healthy
    volumes:
      - ./logstash/pipeline.conf:/usr/share/logstash/pipeline/pipeline.conf:ro
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "9600:9600"
    environment:
      - "LS_JAVA_OPTS=-Xms256m -Xmx256m"
    networks:
      - elk-net
    deploy:
      resources:
        limits:
          memory: 512m

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    depends_on:
      elasticsearch:
        condition: service_healthy
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    networks:
      - elk-net
    deploy:
      resources:
        limits:
          memory: 512m

volumes:
  es-data:

networks:
  elk-net:
    driver: bridge

Important: The deploy.resources.limits.memory keys set memory caps. Elasticsearch is memory-hungry — 1 GB is the minimum for a single-node setup.

Step 3 — Prepare the Host

Elasticsearch requires a higher vm.max_map_count setting on Linux:

# Linux only — not needed on macOS/Docker Desktop
sudo sysctl -w vm.max_map_count=262144

Step 4 — Launch the Stack

docker compose up -d

Note: The ELK stack takes 1–2 minutes to fully start. Elasticsearch must be healthy before Logstash and Kibana connect.

Check status:

docker compose ps

Expected output (after all services are healthy):

NAME                          STATUS              PORTS
elk-compose-elasticsearch-1   Up (healthy)        0.0.0.0:9200->9200/tcp
elk-compose-kibana-1          Up                  0.0.0.0:5601->5601/tcp
elk-compose-logstash-1        Up                  0.0.0.0:5000->5000/tcp, 0.0.0.0:5044->5044/tcp

Step 5 — Test the Pipeline

Send a test log to Logstash via TCP:

echo '{"app":"test","level":"info","message":"Hello from Compose"}' | nc localhost 5000

Verify the log reached Elasticsearch:

curl -s http://localhost:9200/logs-*/_search?pretty | head -30

You should see your test message in the Elasticsearch response.

Step 6 — Access Kibana

Open http://localhost:5601 in your browser.

1. Navigate to Stack Management → Index Patterns
2. Create an index pattern: logs-*
3. Select @timestamp as the time field
4. Go to Discover to see your logs

Enhancements to Try

1. Add Filebeat to collect container logs automatically
2. Add Curator to manage index lifecycle (delete old indices)
3. Add Elastalert for log-based alerting
4. Add resource monitoring with Metricbeat

---

Quick Reference

Item	Detail
Elasticsearch port	9200:9200
Logstash port	5044:5044 (Beats), 5000:5000 (TCP)
Kibana port	5601:5601
ES memory limit	1g
ES health check	curl -f http://localhost:9200/_cluster/health
Pipeline config	./logstash/pipeline.conf