GCP for DevOps — Monitoring Practice Exercises

4 minute read

Monitoring Practice Exercises

Exercise 1 — Create a Custom Dashboard

Create a Cloud Monitoring dashboard that shows CPU utilization for all VMs and request count for Cloud Run services.

Steps:

  1. Create a dashboard JSON with two widgets
  2. Deploy the dashboard
  3. Verify in the Console
Show Answer 
{
  "displayName": "DevOps Overview Dashboard",
  "gridLayout": {
    "columns": 2,
    "widgets": [
      {
        "title": "VM CPU Utilization",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": {
                "filter": "metric.type=\"compute.googleapis.com/instance/cpu/utilization\"",
                "aggregation": {
                  "alignmentPeriod": "60s",
                  "perSeriesAligner": "ALIGN_MEAN"
                }
              }
            },
            "plotType": "LINE"
          }]
        }
      },
      {
        "title": "Cloud Run Requests",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": {
                "filter": "metric.type=\"run.googleapis.com/request_count\"",
                "aggregation": {
                  "alignmentPeriod": "60s",
                  "perSeriesAligner": "ALIGN_RATE"
                }
              }
            },
            "plotType": "STACKED_BAR"
          }]
        }
      }
    ]
  }
}

$ gcloud monitoring dashboards create --config-from-file=dashboard.json
$ gcloud monitoring dashboards list

Exercise 2 — Set Up Alerting

Create an alert policy that triggers when Cloud Run p95 latency exceeds 1 second for 5 minutes, with email notification.

Steps:

  1. Create a notification channel
  2. Create an alert policy with threshold condition
  3. Verify the policy
Show Answer 
# Create notification channel
$ gcloud alpha monitoring channels create \
    --display-name="DevOps Team" \
    --type=email \
    --channel-labels=email_address=devops@example.com

# Note the channel ID from the output

# Create alert policy
$ gcloud alpha monitoring policies create \
    --display-name="Cloud Run High Latency" \
    --condition-display-name="p95 Latency > 1s" \
    --condition-filter='metric.type="run.googleapis.com/request_latencies" AND resource.type="cloud_run_revision"' \
    --condition-threshold-value=1000 \
    --condition-threshold-comparison=COMPARISON_GT \
    --condition-threshold-duration=300s \
    --condition-threshold-aggregation='{"alignmentPeriod":"60s","perSeriesAligner":"ALIGN_PERCENTILE_95"}' \
    --notification-channels="projects/my-project/notificationChannels/CHANNEL_ID" \
    --documentation="p95 latency exceeded 1s. Check Cloud Run logs and scale."

# Verify
$ gcloud alpha monitoring policies list

Exercise 3 — Log Sink to BigQuery

Create a log sink that routes all Cloud Run error logs to BigQuery for analysis.

Steps:

  1. Create a BigQuery dataset
  2. Create a log sink with a filter
  3. Grant the sink’s service account write permission
  4. Verify logs flow to BigQuery
Show Answer 
# Create BigQuery dataset
$ bq mk --dataset --location=us-central1 my-project:run_error_logs

# Create log sink
$ gcloud logging sinks create run-errors-to-bq \
    bigquery.googleapis.com/projects/my-project/datasets/run_error_logs \
    --log-filter='resource.type="cloud_run_revision" AND severity>=ERROR'

# Output shows the writer identity (service account)
# Grant it BigQuery Data Editor role
$ gcloud projects add-iam-policy-binding my-project \
    --member="serviceAccount:WRITER_IDENTITY" \
    --role="roles/bigquery.dataEditor"

# Verify sink
$ gcloud logging sinks describe run-errors-to-bq

# After some errors, query BigQuery
$ bq query --nouse_legacy_sql \
    'SELECT timestamp, jsonPayload.message
     FROM `my-project.run_error_logs.run_googleapis_com_stderr_*`
     ORDER BY timestamp DESC
     LIMIT 20'

Exercise 4 — Custom Metric and Alert

Create a custom metric that tracks order processing count, write data points, and alert when it drops to zero.

Steps:

  1. Create a custom metric descriptor
  2. Write sample data points
  3. Create a metric absence alert
Show Answer 
# create_metric.py
from google.cloud import monitoring_v3
import time

client = monitoring_v3.MetricServiceClient()
project = "projects/my-project"

# Create metric descriptor
descriptor = monitoring_v3.MetricDescriptor()
descriptor.type = "custom.googleapis.com/orders/processed_count"
descriptor.metric_kind = monitoring_v3.MetricDescriptor.MetricKind.GAUGE
descriptor.value_type = monitoring_v3.MetricDescriptor.ValueType.INT64
descriptor.description = "Number of orders processed"

client.create_metric_descriptor(name=project, metric_descriptor=descriptor)
print("Metric descriptor created")

# Write data points
for i in range(5):
    series = monitoring_v3.TimeSeries()
    series.metric.type = "custom.googleapis.com/orders/processed_count"
    series.resource.type = "global"

    now = time.time()
    interval = monitoring_v3.TimeInterval({"end_time": {"seconds": int(now)}})
    point = monitoring_v3.Point({"interval": interval, "value": {"int64_value": 42 + i}})
    series.points = [point]

    client.create_time_series(request={"name": project, "time_series": [series]})
    print(f"Data point {i+1} written")
    time.sleep(65)  # Must wait >1 min between writes for same time series

# Create absence alert (fires when no data for 10 min)
$ gcloud alpha monitoring policies create \
    --display-name="Order Processing Stopped" \
    --condition-display-name="No orders processed" \
    --condition-filter='metric.type="custom.googleapis.com/orders/processed_count"' \
    --condition-threshold-value=0 \
    --condition-threshold-comparison=COMPARISON_GT \
    --condition-threshold-duration=600s \
    --notification-channels="projects/my-project/notificationChannels/CHANNEL_ID"

Exercise 5 — Structured Logging with Trace Correlation

Deploy a Cloud Run service with structured JSON logging that includes trace IDs for correlation.

Steps:

  1. Create a Flask app with structured logging
  2. Include trace context in log entries
  3. Deploy to Cloud Run
  4. Verify logs and traces are correlated in the Console
Show Answer 
# main.py
import json
import os
from flask import Flask, request

app = Flask(__name__)
PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT", "my-project")

def log_structured(severity, message, **kwargs):
    trace_header = request.headers.get("X-Cloud-Trace-Context", "")
    trace_id = trace_header.split("/")[0] if trace_header else ""

    entry = {
        "severity": severity,
        "message": message,
        **kwargs,
    }
    if trace_id:
        entry["logging.googleapis.com/trace"] = f"projects/{PROJECT_ID}/traces/{trace_id}"

    print(json.dumps(entry))

@app.route("/")
def home():
    log_structured("INFO", "Request received",
        httpRequest={"method": request.method, "url": request.url})

    # Simulate processing
    log_structured("INFO", "Processing complete", latency_ms=45)

    return {"status": "ok"}

@app.route("/error")
def error():
    try:
        1 / 0
    except Exception as e:
        import traceback
        log_structured("ERROR", str(e),
            stack_trace=traceback.format_exc(),
            serviceContext={"service": "trace-demo", "version": "1.0"})
        return {"error": str(e)}, 500

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

$ gcloud run deploy trace-demo \
    --source=. \
    --region=us-central1 \
    --allow-unauthenticated

# Test
$ curl https://trace-demo-abc123.a.run.app/
$ curl https://trace-demo-abc123.a.run.app/error

# In Console:
# 1. Go to Cloud Logging — see structured JSON entries
# 2. Click the trace link — opens in Cloud Trace
# 3. Go to Error Reporting — see grouped error

Cleanup

$ gcloud monitoring dashboards delete DASHBOARD_ID --quiet
$ gcloud alpha monitoring policies delete POLICY_ID --quiet
$ gcloud alpha monitoring channels delete CHANNEL_ID --quiet
$ gcloud logging sinks delete run-errors-to-bq --quiet
$ bq rm -r -f my-project:run_error_logs
$ gcloud run services delete trace-demo --region=us-central1 --quiet

Next Chapter → Configuration & Secrets

Updated:

You may also enjoy