ELK Stack — Kubernetes Logging

2 minute read

Kubernetes Logging with the ELK Stack

Kubernetes runs applications as ephemeral containers spread across multiple nodes. When a pod restarts or a node goes down, container logs disappear with it. Centralized logging solves this problem by shipping every container’s output to a durable, searchable backend — and the ELK Stack is one of the most popular choices for that backend.

In this chapter you will learn how to collect, enrich, store, and visualize Kubernetes logs using Elasticsearch, Logstash/Beats, and Kibana.

What You’ll Learn

  • How Kubernetes generates and stores container logs on each node
  • The three standard logging patterns — node-level agent, sidecar container, and direct push
  • Deploying Filebeat as a DaemonSet to collect logs from every node
  • Running Elasticsearch and Kibana inside Kubernetes with the ECK operator
  • How Fluentd and Fluent Bit compare with Logstash and Filebeat
  • Designing log retention and ILM policies for Kubernetes workloads
  • Hands-on exercises that tie every concept together

Chapter Contents

# Topic Description
1 Kubernetes Log Architecture How K8s produces, stores, and rotates container logs
2 Filebeat DaemonSet Deploying Filebeat on every node to ship logs
3 ECK — Elastic Cloud on Kubernetes Running Elasticsearch & Kibana natively on K8s
4 Fluentd and Fluent Bit Lightweight log shippers and the EFK stack
5 Log Retention in Kubernetes ILM, storage planning, and cost optimization
6 Practice Exercises Five hands-on labs with full solutions

Why Centralized Logging Matters in Kubernetes

Traditional applications write logs to files on a persistent server. You can SSH in and tail -f anytime. Kubernetes changes the game:

  • Pods are ephemeral — they are created and destroyed constantly.
  • Nodes scale up and down — auto-scaling removes the machines that hold the logs.
  • Containers restart — the previous container’s stdout is gone after the restart limit.
  • Dozens of microservices — correlating a request across services is impossible without a central index.

A centralized logging pipeline — Filebeat or Fluent Bit on every node, Elasticsearch for storage, Kibana for visualization — gives you a single pane of glass across your entire cluster.

Prerequisites

Before starting this chapter, make sure you are comfortable with:

  • Elasticsearch core concepts (indices, documents, mappings) — covered in earlier chapters
  • Kibana Discover and dashboards
  • Basic Kubernetes concepts (Pods, Deployments, DaemonSets, Namespaces, RBAC)
  • kubectl command-line usage
  • YAML manifest syntax

Cluster requirement: A running Kubernetes cluster (Minikube, kind, or a managed cluster like EKS/GKE/AKS) with kubectl configured.

Let’s begin with Kubernetes Log Architecture to understand how logs flow inside a cluster.

You may also enjoy