KCNA — Logging in Kubernetes

Logging in Kubernetes

Logging is the second pillar of observability. In Kubernetes, containers write logs to stdout/stderr, and a logging infrastructure collects, processes, and stores them centrally.

---

Container Logging Basics

How Kubernetes Handles Logs

1. Containers write to stdout and stderr
2. The container runtime captures these streams
3. Logs are stored on the node’s filesystem (typically /var/log/containers/)
4. kubectl logs reads from these files
5. A logging agent (DaemonSet) ships logs to a central backend

Container (stdout/stderr)
        │
        ▼
Container Runtime (captures streams)
        │
        ▼
Node Filesystem (/var/log/containers/)
        │
        ├──→ kubectl logs (manual access)
        │
        └──→ Logging Agent (DaemonSet)
                │
                ▼
        Central Log Backend
       (Elasticsearch, Loki, etc.)

kubectl logs

# View logs of a pod
kubectl logs my-pod

# View logs of a specific container in a multi-container pod
kubectl logs my-pod -c my-container

# Stream logs in real-time
kubectl logs -f my-pod

# View previous container logs (after crash)
kubectl logs my-pod --previous

# View last 100 lines
kubectl logs my-pod --tail=100

---

Logging Patterns

Pattern 1: Node-Level Agent (DaemonSet)

The most common pattern. A logging agent runs on every node as a DaemonSet.

Node 1                        Node 2
┌──────────────────┐         ┌──────────────────┐
│ ┌───┐ ┌───┐     │         │ ┌───┐ ┌───┐     │
│ │Pod│ │Pod│     │         │ │Pod│ │Pod│     │
│ └─┬─┘ └─┬─┘     │         │ └─┬─┘ └─┬─┘     │
│   └───┬──┘       │         │   └───┬──┘       │
│ ┌─────▼────────┐ │         │ ┌─────▼────────┐ │
│ │ Fluentd/     │ │         │ │ Fluentd/     │ │
│ │ Fluent Bit   │ │         │ │ Fluent Bit   │ │
│ │ (DaemonSet)  │ │         │ │ (DaemonSet)  │ │
│ └──────┬───────┘ │         │ └──────┬───────┘ │
└────────┼─────────┘         └────────┼─────────┘
         └───────────┬────────────────┘
                     ▼
            Central Log Backend

Pattern 2: Sidecar Container

A logging sidecar runs alongside the application container in the same pod.

┌──────────────────────────┐
│          Pod              │
│ ┌───────────┐ ┌────────┐ │
│ │    App    │ │Logging │ │
│ │ Container │→│Sidecar │─┼──→ Log Backend
│ └───────────┘ └────────┘ │
└──────────────────────────┘

Use cases:

• Application writes logs to files (not stdout)
• Need custom log processing per application
• Multi-format log handling

Pattern 3: Direct Push

The application sends logs directly to the logging backend.

Not recommended because:

• Application code is coupled to the logging infrastructure
• Every application must implement logging client logic
• Harder to change the logging backend

---

Logging Tools

Fluentd (CNCF Graduated)

Feature	Details
Type	Log collector and processor
Language	Ruby + C
Plugins	1000+ plugins for inputs, outputs, filters
Buffer	Built-in buffering and retry logic
Use Case	Full-featured log pipeline with complex routing

Fluent Bit (CNCF Graduated — sub-project of Fluentd)

Feature	Details
Type	Lightweight log collector
Language	C
Footprint	~450KB — much smaller than Fluentd
Performance	High throughput, low resource usage
Use Case	Edge, IoT, Kubernetes DaemonSet — collect and forward

Fluentd vs Fluent Bit

Aspect	Fluentd	Fluent Bit
Size	~40MB	~450KB
Plugins	1000+	100+
Processing	Complex routing and transformation	Lightweight collection and forwarding
Typical Role	Aggregator	Collector (DaemonSet on nodes)

Common pattern: Fluent Bit collects on nodes → forwards to Fluentd aggregator → sends to backend.

---

Logging Stacks

EFK Stack (Elasticsearch, Fluentd, Kibana)

Fluent Bit/Fluentd → Elasticsearch → Kibana
(collect)            (store/index)    (visualize/query)

Component	Role
Fluentd/Fluent Bit	Collect and forward logs
Elasticsearch	Store and index logs for fast searching
Kibana	Visualize and query logs

PLG Stack (Promtail, Loki, Grafana)

Promtail → Loki → Grafana
(collect)   (store)  (visualize)

Component	Role
Promtail	Collect and forward logs (like Fluent Bit for Loki)
Loki	Store logs — indexes labels only, not full text (lighter than Elasticsearch)
Grafana	Visualize and query logs (same Grafana used for Prometheus)

Loki is designed for Kubernetes and is much lighter than Elasticsearch. It’s often called “Prometheus, but for logs.”

---

Structured vs Unstructured Logs

Aspect	Structured (JSON)	Unstructured (Plain Text)
Format	{"time":"...","level":"INFO","msg":"Order created"}	2025-02-09 Order created
Querying	Easy — filter by any field	Difficult — requires regex
Parsing	Machine-readable	Requires custom parsers
Best Practice	Preferred in cloud-native	Legacy applications

---

KCNA Exam Tips

• Containers should log to stdout/stderr — not to files (twelve-factor principle)
• Node-Level DaemonSet is the most common logging pattern in Kubernetes
• Know the difference between Fluentd (full-featured) and Fluent Bit (lightweight)
• Both Fluentd and Fluent Bit are CNCF Graduated projects
• Know the two main logging stacks: EFK and PLG
• Loki indexes labels only — it’s lighter than Elasticsearch
• Structured (JSON) logs are preferred over unstructured logs

---

What’s Next?

Explore distributed tracing and OpenTelemetry — understanding request flow across microservices.

Next: Tracing and OpenTelemetry →