GCP for DevOps — Alerting Policies

Alerting Policies

Alerting Policies define conditions that trigger notifications when metrics exceed thresholds. They connect to notification channels (email, Slack, PagerDuty, SMS) to alert on-call teams.

Alerting Architecture

Alerting Flow:
┌──────────┐    ┌──────────────┐    ┌──────────────────┐
│ Metrics   │───▶│  Alerting     │───▶│ Notification      │
│ Store     │    │  Policy       │    │ Channel           │
│           │    │               │    │                   │
│ • CPU     │    │ Condition:    │    │ • Email           │
│ • Memory  │    │ CPU > 80%    │    │ • Slack           │
│ • Latency │    │ for 5 min    │    │ • PagerDuty       │
│ • Errors  │    │               │    │ • SMS             │
└──────────┘    └──────────────┘    │ • Webhook         │
                                     └──────────────────┘

Notification Channels

# Create an email notification channel
$ gcloud alpha monitoring channels create \
    --display-name="DevOps Email" \
    --type=email \
    --channel-labels=email_address=devops@example.com

# Create a Slack notification channel
$ gcloud alpha monitoring channels create \
    --display-name="Slack Alerts" \
    --type=slack \
    --channel-labels=channel_name="#alerts"

# List notification channels
$ gcloud alpha monitoring channels list

# Output:
# NAME                                                DISPLAY_NAME   TYPE   ENABLED
# projects/my-project/notificationChannels/123456     DevOps Email   email  true
# projects/my-project/notificationChannels/789012     Slack Alerts   slack  true

Create Alert Policies via gcloud

# Create a CPU utilization alert
$ gcloud alpha monitoring policies create \
    --display-name="High CPU Alert" \
    --condition-display-name="CPU > 80%" \
    --condition-filter='metric.type="compute.googleapis.com/instance/cpu/utilization" AND resource.type="gce_instance"' \
    --condition-threshold-value=0.8 \
    --condition-threshold-comparison=COMPARISON_GT \
    --condition-threshold-duration=300s \
    --condition-threshold-aggregation='{"alignmentPeriod":"60s","perSeriesAligner":"ALIGN_MEAN"}' \
    --notification-channels="projects/my-project/notificationChannels/123456" \
    --documentation="CPU utilization exceeded 80% for 5 minutes. Check for runaway processes."

Alert Policy JSON

For complex policies, use JSON:

{
  "displayName": "Cloud Run High Latency",
  "combiner": "OR",
  "conditions": [
    {
      "displayName": "p99 Latency > 2s",
      "conditionThreshold": {
        "filter": "metric.type=\"run.googleapis.com/request_latencies\" AND resource.type=\"cloud_run_revision\"",
        "aggregations": [
          {
            "alignmentPeriod": "60s",
            "perSeriesAligner": "ALIGN_PERCENTILE_99"
          }
        ],
        "comparison": "COMPARISON_GT",
        "thresholdValue": 2000,
        "duration": "300s",
        "trigger": {
          "count": 1
        }
      }
    }
  ],
  "notificationChannels": [
    "projects/my-project/notificationChannels/123456"
  ],
  "alertStrategy": {
    "autoClose": "604800s"
  },
  "documentation": {
    "content": "Cloud Run p99 latency exceeded 2 seconds for 5 minutes.\n\n**Runbook:**\n1. Check Cloud Run logs\n2. Verify downstream dependencies\n3. Scale up if needed",
    "mimeType": "text/markdown"
  }
}

# Create from JSON
$ gcloud alpha monitoring policies create \
    --policy-from-file=alert-policy.json

# List alert policies
$ gcloud alpha monitoring policies list

# Delete an alert policy
$ gcloud alpha monitoring policies delete POLICY_ID

Log-Based Alerting

Alert on log entries matching specific patterns:

# Create a log-based alert for errors
$ gcloud alpha monitoring policies create \
    --display-name="Application Errors Alert" \
    --condition-display-name="Error log entries" \
    --condition-filter='metric.type="logging.googleapis.com/user/app_errors"' \
    --condition-threshold-value=10 \
    --condition-threshold-comparison=COMPARISON_GT \
    --condition-threshold-duration=300s \
    --notification-channels="projects/my-project/notificationChannels/123456"

Uptime Check Alerting

Uptime checks automatically create alert policies:

# Create uptime check with alerting
$ gcloud monitoring uptime create api-health \
    --display-name="API Health" \
    --resource-type=uptime-url \
    --hostname=api.example.com \
    --path=/health \
    --check-interval=60s

# The uptime check creates a metric:
# monitoring.googleapis.com/uptime_check/check_passed
# Alert when check fails from multiple regions

Alert Policy Types

Type	Condition	Use Case
Metric Threshold	Value exceeds limit	CPU > 80%, disk > 90%
Metric Absence	No data received	Dead service, broken pipeline
Log-Based	Log entry matches filter	Error logs, security events
Uptime Check	Endpoint unreachable	API down, site offline
Process Health	Process count changes	Critical process died
Forecast	Predicted to exceed	Disk will be full in 24h

Alert Severity and Routing

Alert Routing Strategy:
┌──────────────────┐
│  CRITICAL          │──▶ PagerDuty (wake up on-call)
│  (service down)    │
├──────────────────┤
│  WARNING           │──▶ Slack #alerts (investigate)
│  (degraded)        │
├──────────────────┤
│  INFO              │──▶ Email (review next day)
│  (anomaly)         │
└──────────────────┘

Common Mistakes

Mistake	Why It’s Wrong	Fix
No notification channels	Alert fires, nobody notified	Always configure channels
Too many alerts	Alert fatigue	Focus on actionable alerts
No runbook in documentation	Responder doesn’t know what to do	Add step-by-step runbook
Duration too short	Flapping alerts	Set 5+ minute duration
Missing metric absence alerts	Dead service goes unnoticed	Add absence conditions

Quick Reference

Command	Description
gcloud alpha monitoring policies create	Create alert policy
gcloud alpha monitoring policies list	List policies
gcloud alpha monitoring policies delete	Delete policy
gcloud alpha monitoring channels create	Create notification channel
gcloud alpha monitoring channels list	List channels

Next → Cloud Logging