ELK Stack — Kubernetes Log Architecture
Kubernetes Logging Overview
Every container in Kubernetes writes to two standard streams — stdout and stderr. The container runtime captures these streams and stores them as log files on the node’s filesystem. Understanding where those files live and how they are managed is the foundation for building a centralized logging pipeline.
Container stdout and stderr
When an application inside a container writes to stdout or stderr, the container runtime (containerd, CRI-O) intercepts the output and writes it to a JSON-formatted log file on the host.
# Inside a container
echo "Application started on port 8080" # → stdout
echo "ERROR: connection refused" >&2 # → stderr
The runtime wraps each line with metadata:
{
"log": "Application started on port 8080\n",
"stream": "stdout",
"time": "2026-04-01T09:30:00.123456789Z"
}
Log Location on Nodes
Container logs are stored in two well-known paths on every Kubernetes node:
| Path | Description |
|---|---|
/var/log/containers/ |
Symlinks named <pod>_<namespace>_<container>-<id>.log |
/var/log/pods/<namespace>_<pod>_<uid>/<container>/ |
Actual log files written by the runtime |
# List container log files on a node
ls -la /var/log/containers/
# Example output
# frontend-abc12_default_nginx-a1b2c3.log -> /var/log/pods/default_frontend-abc12_.../nginx/0.log
# backend-xyz99_prod_api-d4e5f6.log -> /var/log/pods/prod_backend-xyz99_.../api/0.log
The symlinks in /var/log/containers/ follow a predictable naming convention that encodes the pod name, namespace, and container name — making them ideal for a log shipper like Filebeat to parse.
Log Rotation
Kubernetes and the container runtime handle log rotation automatically:
- kubelet is responsible for rotating container logs.
- Default rotation: 10 MB per file, 5 rotated files kept.
- Configurable via kubelet flags:
# kubelet log rotation flags
--container-log-max-size=10Mi
--container-log-max-files=5
When a log file exceeds the size limit, it is rotated to 0.log.YYYYMMDD-HHMMSS and a new 0.log is created.
Important: Log rotation means older logs are deleted from the node. Without a centralized logging pipeline, those logs are lost forever.
Logging Patterns
Kubernetes supports three logging patterns. Each has trade-offs in complexity, resource usage, and flexibility.
Pattern 1 — Node-Level Agent
A logging agent (Filebeat, Fluent Bit) runs on every node as a DaemonSet. It reads log files from /var/log/containers/ and ships them to Elasticsearch.
┌──────────────────────────────────────────────────────────────┐
│ Kubernetes Node │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Pod A │ │ Pod B │ │ Pod C │ │
│ │ (nginx) │ │ (api) │ │ (worker) │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ stdout │ stdout │ stdout │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────┐ │
│ │ /var/log/containers/*.log │ │
│ └──────────────────┬──────────────────────┘ │
│ │ reads │
│ ┌──────┴──────┐ │
│ │ Filebeat │ ◄── DaemonSet (1 per node) │
│ │ (agent) │ │
│ └──────┬──────┘ │
│ │ ships │
└─────────────────────┼────────────────────────────────────────┘
▼
┌───────────────┐
│ Elasticsearch │
└───────────────┘
Pros: Simple, low overhead, no changes to application pods. Cons: Limited to stdout/stderr — cannot collect application-specific log files inside the container.
Pattern 2 — Sidecar Container
A sidecar container runs inside the same pod as the application. It reads logs from a shared volume and forwards them.
┌───────────────────────────────────────────────┐
│ Pod │
│ │
│ ┌────────────┐ ┌────────────────┐ │
│ │ App │ │ Sidecar │ │
│ │ Container │ │ (Filebeat / │ │
│ │ │ │ Fluent Bit) │ │
│ └─────┬──────┘ └───────┬────────┘ │
│ │ writes │ reads │
│ ▼ ▼ │
│ ┌─────────────────────────────────────┐ │
│ │ Shared Volume (/var/log/app) │ │
│ └─────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────┘
Sidecar ships logs ──────▶ Elasticsearch
Pros: Can collect any log file, not just stdout/stderr. Per-pod configuration. Cons: Higher resource usage (one sidecar per pod). More complex manifests.
Pattern 3 — Direct Push
The application pushes logs directly to Elasticsearch or a logging API using an HTTP client or SDK.
# Application code pushes directly to Elasticsearch
curl -X POST "http://elasticsearch:9200/app-logs/_doc" \
-H "Content-Type: application/json" \
-d '{"message":"Order processed","orderId":"12345","level":"INFO"}'
Pros: Full control over log format and enrichment. Cons: Tight coupling to the logging backend. Application must handle retries and buffering.
Comparison of Logging Patterns
| Feature | Node Agent | Sidecar | Direct Push |
|---|---|---|---|
| Complexity | Low | Medium | High (code changes) |
| Resource overhead | Low (1 per node) | Medium (1 per pod) | None (in-app) |
| Stdout/stderr | ✅ | ✅ | ❌ (optional) |
| Custom log files | ❌ | ✅ | ✅ |
| Application changes | None | None | Required |
| Most common for | General K8s logging | Legacy apps with file logs | Specialized pipelines |
Recommendation: Start with the node-level agent pattern. It covers 90% of use cases with minimal complexity.
kubectl logs Command
The kubectl logs command reads logs directly from the kubelet — it does not touch Elasticsearch.
# View logs for a pod
kubectl logs frontend-abc12
# Follow logs in real-time
kubectl logs -f frontend-abc12
# View logs for a specific container in a multi-container pod
kubectl logs frontend-abc12 -c nginx
# View previous container's logs (after a restart)
kubectl logs frontend-abc12 --previous
# View logs from the last 30 minutes
kubectl logs frontend-abc12 --since=30m
# View last 100 lines
kubectl logs frontend-abc12 --tail=100
# View logs for all pods with a label
kubectl logs -l app=frontend --all-containers=true
Limitation:
kubectl logsonly shows logs for running (or recently terminated) pods on existing nodes. Once a pod is deleted or a node scales down, the logs are gone — which is exactly why centralized logging exists.
Log Drivers and Container Runtime Interface
The Container Runtime Interface (CRI) defines how the kubelet interacts with the container runtime. Different runtimes handle logging slightly differently:
| Runtime | Log Format | Log Path |
|---|---|---|
| containerd | JSON (CRI log format) | /var/log/pods/... |
| CRI-O | JSON (CRI log format) | /var/log/pods/... |
| Docker (legacy) | JSON-file driver | /var/lib/docker/containers/... |
Since Kubernetes 1.24+, Docker is no longer a supported runtime. containerd and CRI-O both use the CRI log format:
2026-04-01T09:30:00.123456789Z stdout F Application started on port 8080
2026-04-01T09:30:00.234567890Z stderr F ERROR: connection timeout
Format: <timestamp> <stream> <tag> <log message>
F= Full line (complete log entry)P= Partial line (continued on next line, for very long messages)
Quick Reference
| Concept | Details |
|---|---|
| Container log streams | stdout, stderr |
| Node log path | /var/log/containers/, /var/log/pods/ |
| Log rotation defaults | 10 MB per file, 5 files |
| Recommended pattern | Node-level agent (DaemonSet) |
| Common agents | Filebeat, Fluent Bit |
| CRI log format | <timestamp> <stream> <tag> <message> |
| kubectl logs | Reads from kubelet, not centralized |
Next: Filebeat DaemonSet — deploy a node-level logging agent on every Kubernetes node.
