ELK Stack — Project: ELK on Kubernetes with ECK

6 minute read

Project Overview

The Elastic Cloud on Kubernetes (ECK) operator is the official way to deploy and manage Elasticsearch, Kibana, and Beats on Kubernetes. In this project you will:

  • Install the ECK operator
  • Deploy a production-style Elasticsearch cluster
  • Deploy Kibana connected to the cluster
  • Deploy Filebeat as a DaemonSet to collect pod logs
  • Deploy a sample application and route its logs to a dedicated index
  • Build Kibana dashboards for Kubernetes observability

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster                        │
│                                                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                  │
│  │  Node 1  │  │  Node 2  │  │  Node 3  │                  │
│  │┌────────┐│  │┌────────┐│  │┌────────┐│                  │
│  ││Filebeat││  ││Filebeat││  ││Filebeat││  (DaemonSet)     │
│  │└────┬───┘│  │└────┬───┘│  │└────┬───┘│                  │
│  │     │    │  │     │    │  │     │    │                  │
│  │ ┌───┴──┐ │  │ ┌──────┐ │  │ ┌──────┐ │                  │
│  │ │ Pods │ │  │ │ Pods │ │  │ │ Pods │ │                  │
│  │ └──────┘ │  │ └──────┘ │  │ └──────┘ │                  │
│  └──────────┘  └──────────┘  └──────────┘                  │
│                      │                                      │
│                      ▼                                      │
│  ┌──────────────────────────────────────┐                   │
│  │        Elasticsearch Cluster         │                   │
│  │   (3 nodes managed by ECK)           │                   │
│  └─────────────────┬────────────────────┘                   │
│                    │                                        │
│                    ▼                                        │
│  ┌──────────────────────────────────────┐                   │
│  │              Kibana                  │                   │
│  └──────────────────────────────────────┘                   │
│                                                             │
│  ┌──────────────────────────────────────┐                   │
│  │          ECK Operator                │                   │
│  │   (manages all Elastic resources)    │                   │
│  └──────────────────────────────────────┘                   │
└─────────────────────────────────────────────────────────────┘

Prerequisites

  • A running Kubernetes cluster (Minikube, kind, EKS, GKE, or AKS)
  • kubectl configured to access the cluster
  • At least 8 GB of memory available across nodes
  • helm installed (optional, for ECK operator)
# Verify cluster access
kubectl cluster-info
kubectl get nodes

Step 1 — Install the ECK Operator

# Install ECK Custom Resource Definitions
kubectl create -f https://download.elastic.co/downloads/eck/2.12.1/crds.yaml

# Install the ECK operator
kubectl apply -f https://download.elastic.co/downloads/eck/2.12.1/operator.yaml

# Verify the operator is running
kubectl -n elastic-system get pods

Wait until the operator pod shows Running:

kubectl -n elastic-system wait --for=condition=Ready pod -l control-plane=elastic-operator --timeout=120s

Step 2 — Create a Namespace

kubectl create namespace elk-project

Step 3 — Deploy Elasticsearch

Create elasticsearch.yaml:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elk-project
  namespace: elk-project
spec:
  version: 8.13.0
  nodeSets:
    - name: default
      count: 3
      config:
        node.store.allow_mmap: false
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 1Gi
                  cpu: 500m
                limits:
                  memory: 2Gi
                  cpu: "1"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi

Apply it:

kubectl apply -f elasticsearch.yaml

# Watch the deployment progress
kubectl -n elk-project get elasticsearch -w

Wait for the cluster health to become green:

kubectl -n elk-project wait --for=jsonpath='{.status.health}'=green \
  elasticsearch/elk-project --timeout=300s

Retrieve the Elasticsearch password

ES_PASSWORD=$(kubectl -n elk-project get secret elk-project-es-elastic-user \
  -o jsonpath='{.data.elastic}' | base64 -d)
echo "Elasticsearch password: $ES_PASSWORD"

Step 4 — Deploy Kibana

Create kibana.yaml:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: elk-project
  namespace: elk-project
spec:
  version: 8.13.0
  count: 1
  elasticsearchRef:
    name: elk-project
  podTemplate:
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 512Mi
              cpu: 250m
            limits:
              memory: 1Gi
              cpu: 500m

Apply it:

kubectl apply -f kibana.yaml

# Wait for Kibana to be ready
kubectl -n elk-project wait --for=jsonpath='{.status.health}'=green \
  kibana/elk-project --timeout=300s

Access Kibana

# Port-forward to access Kibana locally
kubectl -n elk-project port-forward svc/elk-project-kb-http 5601:5601 &

# Open https://localhost:5601 (accept the self-signed cert)
# Login: elastic / <password from Step 3>

Step 5 — Deploy Filebeat DaemonSet

Create filebeat.yaml:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: elk-project
  namespace: elk-project
spec:
  type: filebeat
  version: 8.13.0
  elasticsearchRef:
    name: elk-project
  config:
    filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
                - logs_path:
                    logs_path: "/var/log/containers/"

    # Route logs based on namespace
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

    output.elasticsearch:
      hosts: ["https://elk-project-es-http:9200"]
      ssl.certificate_authorities:
        - /mnt/elastic/es-certs/ca.crt
      username: elastic
      password: "${ES_PASSWORD}"

    # Index routing by namespace
    setup.template.name: "k8s-logs"
    setup.template.pattern: "k8s-logs-*"

  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        containers:
          - name: filebeat
            env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
              - name: ES_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: elk-project-es-elastic-user
                    key: elastic
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
              - name: es-certs
                mountPath: /mnt/elastic/es-certs
                readOnly: true
            resources:
              requests:
                memory: 128Mi
                cpu: 100m
              limits:
                memory: 256Mi
                cpu: 200m
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers
          - name: es-certs
            secret:
              secretName: elk-project-es-http-certs-public
              optional: false

Create the RBAC resources and apply:

# Create ServiceAccount and ClusterRole for Filebeat
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: elk-project
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces", "nodes"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: elk-project
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
EOF

# Deploy Filebeat
kubectl apply -f filebeat.yaml

# Verify Filebeat pods are running on every node
kubectl -n elk-project get pods -l beat.k8s.elastic.co/name=elk-project

Step 6 — Deploy a Sample Application

Create sample-app.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-generator
  namespace: elk-project
  labels:
    app: log-generator
spec:
  replicas: 2
  selector:
    matchLabels:
      app: log-generator
  template:
    metadata:
      labels:
        app: log-generator
    spec:
      containers:
        - name: log-generator
          image: busybox:1.36
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                LEVEL=$(shuf -n1 -e INFO INFO INFO INFO WARN ERROR)
                MSG=$(shuf -n1 -e "User login" "Page viewed" "Order created" "Payment processed" "DB query slow" "Cache miss" "Connection timeout")
                echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"$LEVEL\",\"app\":\"log-generator\",\"message\":\"$MSG\",\"pod\":\"$HOSTNAME\"}"
                sleep $((RANDOM % 5 + 1))
              done
          resources:
            requests:
              memory: 32Mi
              cpu: 10m
            limits:
              memory: 64Mi
              cpu: 50m
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: elk-project
  labels:
    app: web-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
        - name: nginx
          image: nginx:1.25-alpine
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: 64Mi
              cpu: 50m
            limits:
              memory: 128Mi
              cpu: 100m

Apply it:

kubectl apply -f sample-app.yaml

# Verify pods are running
kubectl -n elk-project get pods -l app=log-generator
kubectl -n elk-project get pods -l app=web-app

Step 7 — Verify Data in Elasticsearch

# Port-forward to Elasticsearch
kubectl -n elk-project port-forward svc/elk-project-es-http 9200:9200 &

# Check indices (use the password from Step 3)
curl -sk -u "elastic:$ES_PASSWORD" https://localhost:9200/_cat/indices?v | grep -E "k8s|filebeat"

Query logs from the sample app

curl -sk -u "elastic:$ES_PASSWORD" "https://localhost:9200/filebeat-*/_search?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "size": 5,
  "query": {
    "match": {
      "kubernetes.labels.app": "log-generator"
    }
  },
  "sort": [{ "@timestamp": "desc" }]
}'

Step 8 — Kibana Dashboards

Access Kibana at https://localhost:5601 and create:

Data View

  • Pattern: filebeat-*
  • Time field: @timestamp

Dashboard Panels

Panel Type Description
Pod Log Volume Stacked bar Breakdown by kubernetes.pod.name
Namespace Activity Pie chart Terms on kubernetes.namespace
Log Level Distribution Donut chart Terms on parsed level field
Error Logs Table Data table Filter level: ERROR or log.level: error
Node Log Distribution Horizontal bar Terms on kubernetes.node.name
Container Restarts Metric Based on container restart events

Step 9 — Namespace-Based Index Routing (Optional)

To route logs from different namespaces to different indices, update the Filebeat config:

# Add to the filebeat config section in filebeat.yaml
output.elasticsearch:
  hosts: ["https://elk-project-es-http:9200"]
  index: "k8s-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}"

setup.template.name: "k8s"
setup.template.pattern: "k8s-*"
setup.ilm.enabled: false

This creates separate indices like k8s-elk-project-2026.04.01 and k8s-default-2026.04.01.

Scaling Considerations

Add dedicated node roles

nodeSets:
  - name: master
    count: 3
    config:
      node.roles: ["master"]
    podTemplate:
      spec:
        containers:
          - name: elasticsearch
            resources:
              limits:
                memory: 2Gi
  - name: data
    count: 3
    config:
      node.roles: ["data", "ingest"]
    podTemplate:
      spec:
        containers:
          - name: elasticsearch
            resources:
              limits:
                memory: 4Gi
    volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

Cleanup

kubectl delete namespace elk-project
kubectl delete clusterrole filebeat
kubectl delete clusterrolebinding filebeat

What You Learned

  • How to install and use the ECK operator
  • How to deploy Elasticsearch and Kibana on Kubernetes
  • How to deploy Filebeat as a DaemonSet for cluster-wide log collection
  • How to use Kubernetes metadata for log enrichment
  • How to route logs by namespace

Next Project

→ Project 4 — Multi-Source Log Aggregation Pipeline

You may also enjoy