ELK Stack — ECK (Elastic Cloud on Kubernetes)

ECK Overview

ECK (Elastic Cloud on Kubernetes) is the official Kubernetes operator from Elastic. It extends the Kubernetes API with Custom Resource Definitions (CRDs) for Elasticsearch, Kibana, Beats, Logstash, and other Elastic components.

Instead of managing StatefulSets, Services, ConfigMaps, and PersistentVolumeClaims yourself, you declare the desired state in a simple YAML manifest and the ECK operator handles the rest — including rolling upgrades, TLS certificate management, and cluster scaling.

---

What ECK Manages

Component	CRD Kind	Description
Elasticsearch	Elasticsearch	Cluster with node sets, storage, TLS
Kibana	Kibana	Deployment with ES connection
Beats	Beat	Filebeat, Metricbeat, etc.
Logstash	Logstash	Pipeline processing
APM Server	ApmServer	Application Performance Monitoring
Elastic Agent	Agent	Unified agent
Elastic Maps	ElasticMapsServer	Geo data

---

Installing the ECK Operator

Step 1 — Install CRDs

# Install ECK CRDs
kubectl create -f https://download.elastic.co/downloads/eck/2.12.1/crds.yaml

Step 2 — Deploy the Operator

# Install the ECK operator into the elastic-system namespace
kubectl apply -f https://download.elastic.co/downloads/eck/2.12.1/operator.yaml

# Verify the operator is running
kubectl get pods -n elastic-system

Expected output:

NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          30s

Step 3 — Verify CRDs

# List ECK custom resource definitions
kubectl get crds | grep elastic

# Expected output
# elasticsearches.elasticsearch.k8s.elastic.co
# kibanas.kibana.k8s.elastic.co
# beats.beat.k8s.elastic.co
# logstashes.logstash.k8s.elastic.co

---

Deploying Elasticsearch with ECK

Minimal Elasticsearch Cluster

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic-system
spec:
  version: 8.13.0
  nodeSets:
    - name: default
      count: 3
      config:
        node.store.allow_mmap: false
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 2Gi
                  cpu: 1
                limits:
                  memory: 2Gi
                  cpu: 2
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 50Gi
            storageClassName: standard

Production Elasticsearch Cluster (Multi-Role)

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic-system
spec:
  version: 8.13.0
  nodeSets:
    # Master nodes
    - name: master
      count: 3
      config:
        node.roles: ["master"]
        node.store.allow_mmap: false
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 2Gi
                  cpu: 500m
                limits:
                  memory: 2Gi
                  cpu: 1
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchLabels:
                      elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
                      elasticsearch.k8s.elastic.co/statefulset-name: elasticsearch-es-master
                  topologyKey: kubernetes.io/hostname
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 10Gi
            storageClassName: standard

    # Data nodes
    - name: data
      count: 3
      config:
        node.roles: ["data", "ingest"]
        node.store.allow_mmap: false
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 4Gi
                  cpu: 1
                limits:
                  memory: 4Gi
                  cpu: 2
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 100Gi
            storageClassName: fast-ssd

# Apply the manifest
kubectl apply -f elasticsearch.yaml

# Watch the deployment
kubectl get elasticsearch -n elastic-system -w

# Check pod status
kubectl get pods -n elastic-system -l elasticsearch.k8s.elastic.co/cluster-name=elasticsearch

Expected output after deployment:

NAME            HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch   green    3       8.13.0    Ready   3m

---

Deploying Kibana with ECK

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elastic-system
spec:
  version: 8.13.0
  count: 1
  elasticsearchRef:
    name: elasticsearch
  podTemplate:
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 1Gi
              cpu: 500m
            limits:
              memory: 1Gi
              cpu: 1

# Apply
kubectl apply -f kibana.yaml

# Check status
kubectl get kibana -n elastic-system

# Expected
# NAME     HEALTH   NODES   VERSION   AGE
# kibana   green    1       8.13.0    60s

---

Deploying Beats with ECK

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: elastic-system
spec:
  type: filebeat
  version: 8.13.0
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
  config:
    filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
                - logs_path:
                    logs_path: "/var/log/containers/"
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        containers:
          - name: filebeat
            securityContext:
              runAsUser: 0
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods

# Apply
kubectl apply -f beat.yaml

# Check status
kubectl get beat -n elastic-system

# Expected
# NAME       HEALTH   AVAILABLE   EXPECTED   TYPE       VERSION   AGE
# filebeat   green    3           3          filebeat   8.13.0    30s

Note: When using the Beat CRD, ECK automatically injects the Elasticsearch connection details (host, credentials, TLS certificates) — you don’t need to configure output.elasticsearch manually.

---

Accessing Elasticsearch

Option 1 — Port Forward

# Port-forward Elasticsearch
kubectl port-forward svc/elasticsearch-es-http -n elastic-system 9200:9200

# Test in another terminal
curl -k https://localhost:9200 -u "elastic:$(kubectl get secret elasticsearch-es-elastic-user -n elastic-system -o jsonpath='{.data.elastic}' | base64 -d)"

Option 2 — Kibana Port Forward

# Port-forward Kibana
kubectl port-forward svc/kibana-kb-http -n elastic-system 5601:5601

# Open in browser
# https://localhost:5601

Option 3 — Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: elastic-system
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - kibana.example.com
      secretName: kibana-tls
  rules:
    - host: kibana.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kibana-kb-http
                port:
                  number: 5601

---

Getting the Elastic Password

ECK automatically generates a password for the elastic superuser and stores it in a Kubernetes Secret.

# Get the elastic user password
kubectl get secret elasticsearch-es-elastic-user \
  -n elastic-system \
  -o go-template='{{.data.elastic | base64decode}}'

---

ECK Resource Configuration

Node Sets

Node sets define groups of Elasticsearch nodes with different roles and hardware profiles.

nodeSets:
  - name: hot
    count: 3
    config:
      node.roles: ["data_hot", "ingest"]
      node.attr.data: "hot"
    volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 200Gi

  - name: warm
    count: 2
    config:
      node.roles: ["data_warm"]
      node.attr.data: "warm"
    volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data
        spec:
          storageClassName: standard-hdd
          resources:
            requests:
              storage: 500Gi

Pod Templates

Customize pod spec for scheduling, tolerations, and init containers.

podTemplate:
  spec:
    initContainers:
      - name: sysctl
        securityContext:
          privileged: true
        command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
    containers:
      - name: elasticsearch
        env:
          - name: ES_JAVA_OPTS
            value: "-Xms1g -Xmx1g"
    tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "elasticsearch"
        effect: "NoSchedule"
    nodeSelector:
      node-type: elasticsearch

Volume Claim Templates

ECK uses volumeClaimTemplates to provision persistent storage via the cluster’s StorageClass.

volumeClaimTemplates:
  - metadata:
      name: elasticsearch-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 100Gi

---

ECK Upgrades

ECK performs rolling upgrades automatically when you change the version field.

# Upgrade Elasticsearch from 8.13.0 to 8.14.0
# Simply update the manifest and re-apply

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic-system
spec:
  version: 8.14.0    # Changed from 8.13.0
  # ... rest of spec unchanged

kubectl apply -f elasticsearch.yaml

# Watch the rolling upgrade
kubectl get pods -n elastic-system -w

ECK upgrades one node at a time, waits for the cluster to become green, then proceeds to the next node.

---

Quick Reference

Concept	Details
ECK operator	Kubernetes operator for Elastic stack
Install CRDs	kubectl create -f .../crds.yaml
Install operator	kubectl apply -f .../operator.yaml
Elasticsearch CRD	kind: Elasticsearch
Kibana CRD	kind: Kibana
Beat CRD	kind: Beat
Default namespace	elastic-system
Password secret	<name>-es-elastic-user
TLS	Enabled by default (self-signed)
Upgrades	Rolling — change version and re-apply

---

Next: Fluentd and Fluent Bit — alternative log shippers and the EFK stack.