GCP for DevOps — Project: Monitoring & Alerting System

4 minute read

Project 5: Monitoring & Alerting System

Build a comprehensive monitoring and alerting system using Cloud Monitoring, Cloud Logging, custom metrics, dashboards, SLOs, and notification channels. This project demonstrates production-grade observability for GCP workloads.

Architecture

Monitoring & Alerting System:
┌──────────────────────────────────────────────────────────┐
│                    GCP Workloads                          │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐   │
│  │Compute  │  │GKE      │  │Cloud Run│  │Cloud SQL│   │
│  │Engine   │  │Cluster  │  │Service  │  │Instance │   │
│  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘   │
│       └─────────────┼───────────┼─────────────┘         │
│                     ▼           ▼                        │
│  ┌──────────────────────────────────────────────┐       │
│  │              Cloud Monitoring                  │       │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐   │       │
│  │  │Dashboards│  │Alerting  │  │SLOs      │   │       │
│  │  │          │  │Policies  │  │          │   │       │
│  │  └──────────┘  └────┬─────┘  └──────────┘   │       │
│  └──────────────────────┼───────────────────────┘       │
│                         │                                │
│              ┌──────────┼──────────┐                    │
│              ▼          ▼          ▼                    │
│        ┌─────────┐ ┌────────┐ ┌─────────┐              │
│        │  Email  │ │ Slack  │ │PagerDuty│              │
│        └─────────┘ └────────┘ └─────────┘              │
└──────────────────────────────────────────────────────────┘

Step 1: Create Notification Channels

# Create an email notification channel
$ gcloud beta monitoring channels create \
    --display-name="DevOps Team Email" \
    --type=email \
    --channel-labels=email_address=devops@example.com
Created notification channel [projects/my-project/notificationChannels/12345].

# Create a Slack notification channel (requires Slack app setup)
# Configure via Console: Monitoring → Alerting → Edit notification channels

# List notification channels
$ gcloud beta monitoring channels list \
    --format="table(name,displayName,type)"
NAME                                                    DISPLAY_NAME        TYPE
projects/my-project/notificationChannels/12345          DevOps Team Email   email

Step 2: Create Alerting Policies

# Alert: High CPU on any VM (> 80% for 5 minutes)
$ gcloud alpha monitoring policies create \
    --display-name="High CPU Alert" \
    --condition-display-name="CPU > 80%" \
    --condition-filter='resource.type="gce_instance" AND metric.type="compute.googleapis.com/instance/cpu/utilization"' \
    --condition-threshold-value=0.8 \
    --condition-threshold-comparison=COMPARISON_GT \
    --condition-threshold-duration=300s \
    --notification-channels="projects/my-project/notificationChannels/12345" \
    --documentation='High CPU detected. Check the instance workload and consider scaling.'
Created alerting policy [projects/my-project/alertPolicies/abc123].

Step 3: Create Custom Metrics

Send custom metrics from your application:

# custom_metrics.py
from google.cloud import monitoring_v3
import time

def create_custom_metric(project_id):
    """Create a custom metric descriptor."""
    client = monitoring_v3.MetricServiceClient()
    project_name = f"projects/{project_id}"
    
    descriptor = monitoring_v3.MetricDescriptor()
    descriptor.type = "custom.googleapis.com/app/request_latency"
    descriptor.metric_kind = monitoring_v3.MetricDescriptor.MetricKind.GAUGE
    descriptor.value_type = monitoring_v3.MetricDescriptor.ValueType.DOUBLE
    descriptor.description = "Application request latency in milliseconds"
    descriptor.unit = "ms"
    
    descriptor = client.create_metric_descriptor(
        name=project_name, metric_descriptor=descriptor
    )
    print(f"Created {descriptor.name}")

def write_metric(project_id, value):
    """Write a data point to the custom metric."""
    client = monitoring_v3.MetricServiceClient()
    project_name = f"projects/{project_id}"
    
    series = monitoring_v3.TimeSeries()
    series.metric.type = "custom.googleapis.com/app/request_latency"
    series.resource.type = "global"
    
    now = time.time()
    interval = monitoring_v3.TimeInterval(
        {"end_time": {"seconds": int(now)}}
    )
    point = monitoring_v3.Point(
        {"interval": interval, "value": {"double_value": value}}
    )
    series.points = [point]
    
    client.create_time_series(
        request={"name": project_name, "time_series": [series]}
    )
    print(f"Wrote metric value: {value}")

Step 4: Create Log-Based Metrics

# Create a log-based metric for error count
$ gcloud logging metrics create error-count \
    --description="Count of error log entries" \
    --log-filter='severity>=ERROR'
Created [error-count].

# Create a log-based metric for specific application errors
$ gcloud logging metrics create app-500-errors \
    --description="Count of HTTP 500 errors" \
    --log-filter='resource.type="cloud_run_revision" AND httpRequest.status=500'
Created [app-500-errors].

# List log-based metrics
$ gcloud logging metrics list \
    --format="table(name,description,filter)"
NAME              DESCRIPTION                    FILTER
error-count       Count of error log entries     severity>=ERROR
app-500-errors    Count of HTTP 500 errors       resource.type="cloud_run_revision"...

Step 5: Create a Dashboard (JSON Definition)

$ cat > dashboard.json << 'ENDJSON'
{
  "displayName": "Application Overview",
  "gridLayout": {
    "columns": 2,
    "widgets": [
      {
        "title": "CPU Utilization",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": {
                "filter": "resource.type=\"gce_instance\" AND metric.type=\"compute.googleapis.com/instance/cpu/utilization\"",
                "aggregation": {
                  "alignmentPeriod": "60s",
                  "perSeriesAligner": "ALIGN_MEAN"
                }
              }
            }
          }]
        }
      },
      {
        "title": "Memory Usage",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": {
                "filter": "resource.type=\"gce_instance\" AND metric.type=\"agent.googleapis.com/memory/percent_used\"",
                "aggregation": {
                  "alignmentPeriod": "60s",
                  "perSeriesAligner": "ALIGN_MEAN"
                }
              }
            }
          }]
        }
      },
      {
        "title": "Error Rate (log-based)",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": {
                "filter": "metric.type=\"logging.googleapis.com/user/error-count\"",
                "aggregation": {
                  "alignmentPeriod": "300s",
                  "perSeriesAligner": "ALIGN_RATE"
                }
              }
            }
          }]
        }
      },
      {
        "title": "Request Latency (custom)",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": {
                "filter": "metric.type=\"custom.googleapis.com/app/request_latency\"",
                "aggregation": {
                  "alignmentPeriod": "60s",
                  "perSeriesAligner": "ALIGN_MEAN"
                }
              }
            }
          }]
        }
      }
    ]
  }
}
ENDJSON

$ gcloud monitoring dashboards create --config-from-file=dashboard.json
Created dashboard [projects/my-project/dashboards/abc123].

Step 6: Define SLOs

# Create a Cloud Run service for SLO monitoring
# First, create a custom service
$ gcloud monitoring services create items-api-service \
    --display-name="Items API Service"
Created service [items-api-service].

# Create an availability SLO (99.9% uptime)
$ cat > slo-availability.json << 'ENDJSON'
{
  "displayName": "Availability SLO - 99.9%",
  "serviceLevelIndicator": {
    "basicSli": {
      "availability": {}
    }
  },
  "goal": 0.999,
  "calendarPeriod": "MONTH"
}
ENDJSON

$ gcloud monitoring slos create \
    --service=items-api-service \
    --config-from-file=slo-availability.json
Created SLO [projects/my-project/services/items-api-service/serviceLevelObjectives/xyz].

Step 7: Create Uptime Checks

# Create an uptime check for a public endpoint
$ gcloud monitoring uptime create \
    --display-name="Items API Health Check" \
    --resource-type=uptime-url \
    --hostname=items-api-abc123-uc.a.run.app \
    --path=/health \
    --check-interval=60 \
    --timeout=10 \
    --regions=USA,EUROPE,ASIA_PACIFIC
Created uptime check [projects/my-project/uptimeCheckConfigs/abc123].

# List uptime checks
$ gcloud monitoring uptime list-configs \
    --format="table(displayName,monitoredResource.labels.host,period)"

Step 8: Alert on SLO Burn Rate

Create an alert that fires when your error budget is burning too fast:

# Alert when burn rate exceeds 10x (fast burn)
$ gcloud alpha monitoring policies create \
    --display-name="SLO Fast Burn Alert" \
    --condition-display-name="Error budget fast burn" \
    --condition-filter='select_slo_burn_rate("projects/my-project/services/items-api-service/serviceLevelObjectives/xyz", "60m")' \
    --condition-threshold-value=10 \
    --condition-threshold-comparison=COMPARISON_GT \
    --condition-threshold-duration=0s \
    --notification-channels="projects/my-project/notificationChannels/12345" \
    --documentation='SLO error budget burning fast. Investigate immediately.'

Monitoring Checklist

Component	Metric	Alert Threshold
CPU utilization	`compute.googleapis.com/instance/cpu/utilization`	> 80% for 5 min
Memory usage	`agent.googleapis.com/memory/percent_used`	> 90% for 5 min
Disk usage	`agent.googleapis.com/disk/percent_used`	> 85% for 10 min
Error rate	`logging.googleapis.com/user/error-count`	> 10/min for 5 min
Uptime	Uptime check failure	Any failure for 2 checks
SLO burn rate	SLO burn rate	> 10x for 1 hour

Clean Up

# Delete dashboards
$ gcloud monitoring dashboards delete projects/my-project/dashboards/abc123

# Delete alerting policies
$ gcloud alpha monitoring policies delete projects/my-project/alertPolicies/abc123

# Delete notification channels
$ gcloud beta monitoring channels delete projects/my-project/notificationChannels/12345

# Delete uptime checks
$ gcloud monitoring uptime delete projects/my-project/uptimeCheckConfigs/abc123

# Delete SLOs and services
$ gcloud monitoring slos delete projects/my-project/services/items-api-service/serviceLevelObjectives/xyz
$ gcloud monitoring services delete items-api-service

# Delete log-based metrics
$ gcloud logging metrics delete error-count
$ gcloud logging metrics delete app-500-errors

Key Learnings

Custom metrics for application-specific monitoring
Log-based metrics for error tracking
Dashboard creation via JSON definitions
SLOs and error budget alerting
Multi-channel notification (email, Slack, PagerDuty)
Uptime checks for external endpoint monitoring

Next → Practice Exercises

Share on

Twitter Facebook LinkedIn