ELK Stack — Project: Centralized Nginx Log Monitoring

Project Overview

In this project you will build a complete centralized logging pipeline for Nginx. By the end you will have:

• An Nginx web server generating access and error logs
• Filebeat shipping those logs to Logstash
• Logstash parsing Nginx log formats with grok
• Elasticsearch storing the structured data
• Kibana dashboards visualizing traffic patterns, response codes, and errors

This is one of the most common real-world ELK use cases and a great foundation for every other project in this chapter.

---

Architecture

┌─────────────┐     ┌───────────┐     ┌───────────┐     ┌───────────────┐     ┌─────────┐
│   Nginx     │────▶│ Filebeat  │────▶│ Logstash  │────▶│ Elasticsearch │────▶│ Kibana  │
│  (logs)     │     │ (shipper) │     │ (parser)  │     │   (storage)   │     │ (viz)   │
└─────────────┘     └───────────┘     └───────────┘     └───────────────┘     └─────────┘
   access.log          tail &            grok &             index:              dashboards
   error.log           ship             enrich          nginx-access-*
                                                        nginx-error-*

---

Project Structure

nginx-elk-project/
├── docker-compose.yml
├── nginx/
│   ├── nginx.conf
│   └── html/
│       └── index.html
├── filebeat/
│   └── filebeat.yml
├── logstash/
│   └── pipeline/
│       └── nginx.conf
└── README.md

---

Step 1 — Docker Compose Setup

Create docker-compose.yml:

version: "3.8"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    container_name: es01
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    networks:
      - elk
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 10

  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    container_name: logstash
    volumes:
      - ./logstash/pipeline/nginx.conf:/usr/share/logstash/pipeline/nginx.conf:ro
    ports:
      - "5044:5044"
    environment:
      - "LS_JAVA_OPTS=-Xms256m -Xmx256m"
    networks:
      - elk
    depends_on:
      elasticsearch:
        condition: service_healthy

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://es01:9200
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      elasticsearch:
        condition: service_healthy

  nginx:
    image: nginx:1.25-alpine
    container_name: nginx-web
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/html:/usr/share/nginx/html:ro
      - nginx-logs:/var/log/nginx
    ports:
      - "8080:80"
    networks:
      - elk

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.13.0
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - nginx-logs:/var/log/nginx:ro
    networks:
      - elk
    depends_on:
      - logstash
      - nginx

volumes:
  es-data:
  nginx-logs:

networks:
  elk:
    driver: bridge

---

Step 2 — Nginx Configuration

Create nginx/nginx.conf:

worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid       /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    # Combined log format (default)
    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

    access_log /var/log/nginx/access.log main;

    sendfile on;
    keepalive_timeout 65;

    server {
        listen 80;
        server_name localhost;

        location / {
            root /usr/share/nginx/html;
            index index.html;
        }

        # Intentional 404 endpoint for testing error logs
        location /generate-error {
            return 404 "Not Found - Test Error";
        }

        # Health check
        location /health {
            return 200 "OK";
        }
    }
}

Create nginx/html/index.html:

<!DOCTYPE html>
<html>
<head><title>Nginx ELK Project</title></head>
<body>
<h1>Welcome to the Nginx ELK Logging Project</h1>
<p>This page generates access logs shipped to the ELK Stack.</p>
</body>
</html>

---

Step 3 — Filebeat Configuration

Create filebeat/filebeat.yml:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      log_type: nginx_access
    fields_under_root: true

  - type: log
    enabled: true
    paths:
      - /var/log/nginx/error.log
    fields:
      log_type: nginx_error
    fields_under_root: true

output.logstash:
  hosts: ["logstash:5044"]

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 3

---

Step 4 — Logstash Pipeline

Create logstash/pipeline/nginx.conf:

input {
  beats {
    port => 5044
  }
}

filter {
  if [log_type] == "nginx_access" {
    grok {
      match => {
        "message" => '%{IPORHOST:client_ip} - %{DATA:remote_user} \[%{HTTPDATE:timestamp}\] "%{WORD:http_method} %{URIPATHPARAM:request_uri} HTTP/%{NUMBER:http_version}" %{NUMBER:response_code:integer} %{NUMBER:bytes_sent:integer} "%{DATA:referrer}" "%{DATA:user_agent}"'
      }
      remove_field => ["message"]
    }

    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"
      remove_field => ["timestamp"]
    }

    geoip {
      source => "client_ip"
      target => "geoip"
    }

    useragent {
      source => "user_agent"
      target => "ua"
    }

    mutate {
      add_field => { "[@metadata][index]" => "nginx-access" }
    }
  }

  else if [log_type] == "nginx_error" {
    grok {
      match => {
        "message" => '%{DATA:timestamp} \[%{LOGLEVEL:log_level}\] %{POSINT:pid}#%{NUMBER:tid}: (\*%{NUMBER:connection_id} )?%{GREEDYDATA:error_message}'
      }
      remove_field => ["message"]
    }

    mutate {
      add_field => { "[@metadata][index]" => "nginx-error" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://es01:9200"]
    index => "%{[@metadata][index]}-%{+YYYY.MM.dd}"
  }

  # Uncomment for debugging
  # stdout { codec => rubydebug }
}

---

Step 5 — Launch the Stack

# Start all services
docker-compose up -d

# Verify all containers are running
docker-compose ps

# Wait for Elasticsearch to be healthy
until curl -s http://localhost:9200/_cluster/health | grep -q '"status":"green"\|"status":"yellow"'; do
  echo "Waiting for Elasticsearch..."
  sleep 5
done

echo "Elasticsearch is ready!"

---

Step 6 — Generate Test Traffic

# Generate access logs
for i in $(seq 1 100); do
  curl -s http://localhost:8080/ > /dev/null
  curl -s http://localhost:8080/health > /dev/null
  curl -s http://localhost:8080/generate-error > /dev/null 2>&1
  curl -s http://localhost:8080/nonexistent > /dev/null 2>&1
done

echo "Generated 400 requests"

---

Step 7 — Verify the Pipeline

Check Elasticsearch indices

curl -s http://localhost:9200/_cat/indices?v | grep nginx

Expected output:

green open nginx-access-2026.04.01 ... 300 0
green open nginx-error-2026.04.01  ... ...

Query sample documents

curl -s http://localhost:9200/nginx-access-*/_search?pretty -H 'Content-Type: application/json' -d'
{
  "size": 1,
  "query": { "match_all": {} }
}'

Check response code breakdown

curl -s http://localhost:9200/nginx-access-*/_search?pretty -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "aggs": {
    "response_codes": {
      "terms": { "field": "response_code" }
    }
  }
}'

---

Step 8 — Kibana Dashboard Plan

Open Kibana at http://localhost:5601 and create the following visualizations.

Create Data View

1. Go to Stack Management → Data Views
2. Create a data view for nginx-access-*
3. Set the time field to @timestamp

Dashboard Panels

Panel	Type	Description
Request Count Over Time	Line chart	@timestamp on X-axis, count on Y-axis
Top 10 URLs	Horizontal bar	Terms aggregation on request_uri
Response Code Distribution	Pie chart	Terms aggregation on response_code
HTTP Methods	Donut chart	Terms aggregation on http_method
Error Rate Over Time	Area chart	Filter response_code >= 400, date histogram
Geographic Map	Map visualization	Based on geoip.location field
Top User Agents	Tag cloud	Terms aggregation on ua.name
Bytes Sent Over Time	Area chart	Date histogram with sum of bytes_sent

Sample Kibana Saved Search (KQL)

response_code >= 400 AND request_uri : "/generate-error"

---

Troubleshooting

Symptom	Cause	Fix
No data in Elasticsearch	Filebeat cannot reach Logstash	Check docker logs filebeat
Grok parse failures	Log format mismatch	Check _grokparsefailure tag in documents
Kibana shows no results	Wrong time range selected	Adjust time picker to cover ingestion window
Elasticsearch OOM	Insufficient memory	Increase ES_JAVA_OPTS or Docker memory limit

---

Cleanup

docker-compose down -v

---

What You Learned

• How to configure Nginx for centralized logging
• How to ship logs with Filebeat to Logstash
• How to parse Nginx log formats with Logstash grok patterns
• How to enrich logs with GeoIP and user-agent parsing
• How to build meaningful Kibana dashboards

---

Next Project

→ Project 2 — Microservices Centralized Logging