KCNA — Tracing and OpenTelemetry

4 minute read

Tracing and OpenTelemetry

Distributed tracing is the third pillar of observability. When a single user request passes through multiple microservices, traces help you understand where time is spent and where failures occur.

Why Tracing?

In a monolith, debugging is straightforward — one process, one log file. In microservices, a single request may touch 10+ services.

User → API Gateway → Auth → Order → Inventory → Payment → Notification

Without tracing:

  • “The checkout is slow” → But which service is slow?
  • “An order failed” → But where in the chain did it fail?

With tracing:

  • See the complete request path with timing for each service
  • Identify the exact service causing latency or errors

Key Concepts

Trace

A trace represents the complete journey of a request through the system. It has a unique Trace ID that connects all related work.

Span

A span represents a single unit of work within a trace. Each service call creates a span.

Span Property Description
Trace ID Unique ID shared by all spans in the trace
Span ID Unique ID for this specific span
Parent Span ID The span that initiated this span
Operation Name What the span represents (e.g., “HTTP GET /orders”)
Start/End Time Duration of the operation
Tags/Attributes Metadata (HTTP status, error flag, etc.)
Logs/Events Timestamped events within the span

Trace Visualization

Trace ID: abc-123
├── API Gateway         ─────────────────────── 120ms
│   ├── Auth Service    ────── 15ms
│   └── Order Service   ──────────────── 95ms
│       ├── Inventory   ───── 20ms
│       └── Payment     ───────── 45ms
│           └── Bank API ──── 30ms

Context Propagation

For tracing to work across services, the trace context must be passed (propagated) between services.

Service A                    Service B                   Service C
┌──────────┐                ┌──────────┐                ┌──────────┐
│ Span A   │  ── headers ──→│ Span B   │  ── headers ──→│ Span C   │
│ trace=abc│  traceparent   │ trace=abc│  traceparent   │ trace=abc│
└──────────┘                └──────────┘                └──────────┘

Context is propagated via HTTP headers:

  • W3C Trace Context: traceparent header (standard)
  • B3: Used by Zipkin
  • Jaeger: Custom headers

Jaeger (CNCF Graduated)

Jaeger is a distributed tracing platform, originally developed at Uber.

Architecture

Application ──→ Jaeger Agent ──→ Jaeger Collector ──→ Storage ──→ Jaeger Query/UI
(SDK/client)    (sidecar/daemon)  (processes traces)    (ES/Cassandra)  (visualization)
Component Purpose
Jaeger Client SDK in your application for creating spans
Jaeger Agent Receives spans locally, batches and sends to collector
Jaeger Collector Validates, indexes, and stores traces
Storage Elasticsearch, Cassandra, or Kafka
Jaeger Query + UI Web interface for searching and visualizing traces

Key Features

  • Distributed context propagation across services
  • Performance and latency optimization — find slow services
  • Root cause analysis — trace errors through the call chain
  • Service dependency analysis — visualize service relationships

OpenTelemetry (CNCF Incubating)

OpenTelemetry (OTel) is the emerging standard for collecting all three pillars of observability data — metrics, logs, and traces — through a single framework.

What OpenTelemetry Provides

Component Purpose
APIs Standard interfaces for instrumentation
SDKs Language-specific implementations (Go, Java, Python, JS, etc.)
Collector Agent that receives, processes, and exports telemetry data
OTLP OpenTelemetry Protocol — standard wire format

OpenTelemetry Collector

┌─────────┐   ┌─────────┐   ┌─────────┐
│ App 1   │   │ App 2   │   │ App 3   │
│ (OTel   │   │ (OTel   │   │ (OTel   │
│  SDK)   │   │  SDK)   │   │  SDK)   │
└────┬────┘   └────┬────┘   └────┬────┘
     │              │              │
     └──────────────┼──────────────┘
                    ▼
     ┌──────────────────────────┐
     │  OpenTelemetry Collector  │
     │  ┌──────────────────────┐│
     │  │     Receivers        ││  ← Receive data (OTLP, Jaeger, Zipkin)
     │  ├──────────────────────┤│
     │  │     Processors       ││  ← Transform, filter, batch
     │  ├──────────────────────┤│
     │  │     Exporters        ││  ← Send to backends
     │  └──────────────────────┘│
     └──────────────────────────┘
                    │
         ┌──────────┼──────────┐
         ▼          ▼          ▼
     Jaeger    Prometheus    Loki
    (traces)   (metrics)    (logs)

Why OpenTelemetry Matters

Before OTel After OTel
Separate SDKs for metrics (Prometheus client), traces (Jaeger client), logs (custom) One unified SDK for all signals
Vendor-specific formats Standard OTLP format
Hard to switch backends Change exporter config, keep instrumentation
Fragmented ecosystem Converged standard

OpenTelemetry merged OpenTracing and OpenCensus into a single project.

Sampling

In high-traffic systems, collecting every trace is expensive. Sampling reduces the volume:

Strategy Description
Head-based Decide to sample at the start of the trace (e.g., 10% of all requests)
Tail-based Decide after the trace completes — sample error traces and slow traces
Rate limiting Collect a fixed number of traces per second

Tail-based sampling is preferred because it captures interesting traces (errors, slow requests) while dropping routine ones.

KCNA Exam Tips

  • Know what a trace and span are — a trace is the full journey, spans are individual steps
  • Context propagation passes trace IDs between services via HTTP headers
  • Jaeger is a CNCF Graduated distributed tracing platform
  • OpenTelemetry is the unified standard for metrics, logs, and traces (CNCF Incubating)
  • OpenTelemetry merged OpenTracing and OpenCensus
  • The OTel Collector has three stages: Receivers → Processors → Exporters
  • Know the difference between head-based and tail-based sampling

What’s Next?

Learn about cost management in cloud-native environments — FinOps practices and resource optimization.

Next: Cost Management →

Updated:

You may also enjoy