GCP for DevOps — Cloud Trace

Cloud Trace

Cloud Trace is a distributed tracing system that collects latency data from applications. It helps you understand how requests propagate across microservices and identify performance bottlenecks.

How Tracing Works

Distributed Trace:
┌───────────────────────────────────────────────────────┐
│ Trace ID: abc123                                       │
│                                                        │
│ ├─ Span: API Gateway          [0ms ──────── 250ms]    │
│ │  ├─ Span: Auth Service      [10ms ── 50ms]          │
│ │  ├─ Span: User Service      [55ms ──────── 200ms]   │
│ │  │  ├─ Span: DB Query       [60ms ── 100ms]         │
│ │  │  └─ Span: Cache Lookup   [105ms ── 115ms]        │
│ │  └─ Span: Response Build    [205ms ── 240ms]        │
└───────────────────────────────────────────────────────┘

Auto-Instrumentation

Cloud Run, Cloud Functions, and App Engine automatically collect traces. For custom apps, use OpenTelemetry:

# pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-gcp-trace
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter

# Set up tracing
provider = TracerProvider()
processor = BatchSpanProcessor(CloudTraceSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

# Create spans
with tracer.start_as_current_span("process-order") as span:
    span.set_attribute("order.id", "order-123")
    span.set_attribute("order.items", 5)

    with tracer.start_as_current_span("validate-payment"):
        # Payment validation logic
        pass

    with tracer.start_as_current_span("update-inventory"):
        # Inventory update logic
        pass

Flask Integration

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from flask import Flask

app = Flask(__name__)

# Auto-instrument Flask and outgoing HTTP requests
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

@app.route("/api/users/<user_id>")
def get_user(user_id):
    # This request is automatically traced
    # Child spans are created for downstream calls
    return {"user_id": user_id, "name": "Alice"}

Viewing Traces

# List recent traces
$ gcloud traces list --limit=10

# Get a specific trace
$ gcloud traces describe TRACE_ID

In the Cloud Console:

1. Navigate to Trace > Trace list
2. Filter by service, latency, or time range
3. Click a trace to see the waterfall view
4. Identify slow spans and bottlenecks

Trace Analysis

Metric	Description	Action
p50 Latency	Median request time	Baseline performance
p95 Latency	95th percentile	Tail latency issues
p99 Latency	99th percentile	Worst-case experience
Span Count	Spans per trace	Over-instrumentation check
Error Rate	Failed spans	Service health

Sampling Configuration

For high-traffic services, configure sampling to control costs:

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

# Sample 10% of traces
sampler = TraceIdRatioBased(0.1)
provider = TracerProvider(sampler=sampler)

Correlate Traces with Logs

Link traces to log entries for full observability:

import json
from opentelemetry import trace

def log_with_trace(severity, message, **kwargs):
    span = trace.get_current_span()
    ctx = span.get_span_context()
    entry = {
        "severity": severity,
        "message": message,
        "logging.googleapis.com/trace": f"projects/my-project/traces/{format(ctx.trace_id, '032x')}",
        "logging.googleapis.com/spanId": format(ctx.span_id, '016x'),
        **kwargs,
    }
    print(json.dumps(entry))

Common Mistakes

Mistake	Why It’s Wrong	Fix
No sampling for high traffic	Excessive costs	Use ratio-based sampling
Missing context propagation	Broken trace chains	Pass trace headers between services
No custom span attributes	Limited debugging info	Add order IDs, user IDs, etc.
Not correlating with logs	Can’t find logs for a trace	Include trace ID in log entries
Over-instrumenting	Too many spans per trace	Trace meaningful operations only

Quick Reference

Command	Description
gcloud traces list	List recent traces
gcloud traces describe TRACE_ID	View trace details
Console: Trace > Trace list	Visual trace explorer
Console: Trace > Analysis reports	Latency analysis

Next → Error Reporting & Profiler