AWS for DevOps — Project 5: Monitoring & Alerting System

Project Overview

Build a comprehensive monitoring and alerting system using CloudWatch, SNS, Lambda, and EventBridge. Set up dashboards, alarms, automated notifications, and self-healing automation that responds to infrastructure events.

Real-World Analogy: You’re building the security monitoring center for a building — cameras (metrics), alarm systems (CloudWatch Alarms), a phone tree (SNS notifications), and automated door locks (Lambda auto-remediation) that respond to intrusions without human intervention.

Detail	Value
Difficulty	⭐⭐⭐ (Advanced)
Time	2-3 hours
Cost	~$0 (mostly within Free Tier)
Services	CloudWatch, SNS, Lambda, EventBridge, EC2

---

Architecture

┌──────────────────────────────────────────────────────────────┐
│               Monitoring & Alerting System                    │
│                                                              │
│  ┌──────────┐   ┌──────────────┐   ┌─────────────────────┐ │
│  │CloudWatch│──→│   Alarms      │──→│       SNS           │ │
│  │ Metrics  │   │ (thresholds)  │   │  ┌──────────────┐  │ │
│  └──────────┘   └──────────────┘   │  │ Email Notify  │  │ │
│                                     │  │ Slack Webhook │  │ │
│  ┌──────────┐   ┌──────────────┐   │  │ Lambda Func   │  │ │
│  │CloudWatch│──→│ Log Metric   │   │  └──────────────┘  │ │
│  │  Logs    │   │   Filters    │   └─────────────────────┘ │
│  └──────────┘   └──────────────┘                            │
│                                                              │
│  ┌──────────────┐   ┌──────────────────────────────────┐   │
│  │ EventBridge  │──→│  Lambda Auto-Remediation          │   │
│  │  Rules       │   │  - Restart unhealthy instances    │   │
│  │              │   │  - Isolate compromised resources  │   │
│  │  Events:     │   │  - Scale up on high utilization   │   │
│  │  - EC2 state │   └──────────────────────────────────┘   │
│  │  - Health    │                                           │
│  │  - GuardDuty │   ┌──────────────────────────────────┐   │
│  └──────────────┘   │  CloudWatch Dashboard             │   │
│                      │  - CPU, Memory, Disk              │   │
│                      │  - Request count, Latency         │   │
│                      │  - Error rates, Alarm status      │   │
│                      └──────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────┘

---

Step 1 — Create SNS Topics for Notifications

# Create notification topics
$ aws sns create-topic --name critical-alerts \
    --tags Key=Project,Value=practice-5
{
    "TopicArn": "arn:aws:sns:us-east-1:123456789012:critical-alerts"
}

$ aws sns create-topic --name warning-alerts \
    --tags Key=Project,Value=practice-5
{
    "TopicArn": "arn:aws:sns:us-east-1:123456789012:warning-alerts"
}

$ aws sns create-topic --name info-alerts \
    --tags Key=Project,Value=practice-5

# Subscribe your email
$ aws sns subscribe \
    --topic-arn arn:aws:sns:us-east-1:123456789012:critical-alerts \
    --protocol email \
    --notification-endpoint your-email@example.com

# Confirm the subscription (check your email and click confirm)

# Verify subscription
$ aws sns list-subscriptions-by-topic \
    --topic-arn arn:aws:sns:us-east-1:123456789012:critical-alerts

Notification Tiers

Tier	Topic	Use Case	Response Time
Critical	critical-alerts	Service down, security breach	Immediate
Warning	warning-alerts	High CPU, disk filling up	Within 1 hour
Info	info-alerts	Deployments, scaling events	Next business day

---

Step 2 — Create CloudWatch Alarms

CPU Utilization Alarm

# High CPU alarm (Warning at 70%, Critical at 90%)
$ aws cloudwatch put-metric-alarm \
    --alarm-name "EC2-HighCPU-Warning" \
    --alarm-description "EC2 CPU utilization above 70%" \
    --namespace AWS/EC2 \
    --metric-name CPUUtilization \
    --statistic Average \
    --period 300 \
    --threshold 70 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 3 \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:warning-alerts \
    --ok-actions arn:aws:sns:us-east-1:123456789012:info-alerts \
    --dimensions Name=InstanceId,Value=i-0abc123 \
    --tags Key=Project,Value=practice-5

$ aws cloudwatch put-metric-alarm \
    --alarm-name "EC2-HighCPU-Critical" \
    --alarm-description "EC2 CPU utilization above 90%" \
    --namespace AWS/EC2 \
    --metric-name CPUUtilization \
    --statistic Average \
    --period 300 \
    --threshold 90 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 2 \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:critical-alerts \
    --dimensions Name=InstanceId,Value=i-0abc123

ALB Alarm — High Error Rate

$ aws cloudwatch put-metric-alarm \
    --alarm-name "ALB-High5xxRate" \
    --alarm-description "ALB 5xx error rate above 5%" \
    --namespace AWS/ApplicationELB \
    --metric-name HTTPCode_Target_5XX_Count \
    --statistic Sum \
    --period 60 \
    --threshold 10 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 3 \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:critical-alerts \
    --dimensions Name=LoadBalancer,Value=app/prod-alb/abc123

RDS Alarm — Low Free Storage

$ aws cloudwatch put-metric-alarm \
    --alarm-name "RDS-LowStorage" \
    --alarm-description "RDS free storage below 2GB" \
    --namespace AWS/RDS \
    --metric-name FreeStorageSpace \
    --statistic Average \
    --period 300 \
    --threshold 2000000000 \
    --comparison-operator LessThanThreshold \
    --evaluation-periods 1 \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:critical-alerts \
    --dimensions Name=DBInstanceIdentifier,Value=prod-db

Billing Alarm

$ aws cloudwatch put-metric-alarm \
    --alarm-name "Billing-Exceeds-50USD" \
    --alarm-description "Estimated charges exceed $50" \
    --namespace AWS/Billing \
    --metric-name EstimatedCharges \
    --statistic Maximum \
    --period 21600 \
    --threshold 50 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 1 \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:critical-alerts \
    --dimensions Name=Currency,Value=USD \
    --region us-east-1

Alarms Summary

Alarm	Metric	Threshold	Severity
EC2-HighCPU-Warning	CPUUtilization	>70% (3 periods)	Warning
EC2-HighCPU-Critical	CPUUtilization	>90% (2 periods)	Critical
ALB-High5xxRate	5XX Count	>10/min (3 periods)	Critical
RDS-LowStorage	FreeStorageSpace	<2GB	Critical
Billing-Exceeds-50USD	EstimatedCharges	>$50	Critical

---

Step 3 — Create Log Metric Filters

# Create a metric filter for application errors
$ aws logs put-metric-filter \
    --log-group-name /ecs/fargate-app \
    --filter-name "ErrorCount" \
    --filter-pattern "ERROR" \
    --metric-transformations \
        metricName=ApplicationErrors,metricNamespace=Custom/App,metricValue=1,defaultValue=0

# Create alarm on the custom metric
$ aws cloudwatch put-metric-alarm \
    --alarm-name "App-HighErrorRate" \
    --namespace Custom/App \
    --metric-name ApplicationErrors \
    --statistic Sum \
    --period 300 \
    --threshold 50 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 1 \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:warning-alerts

# Create metric filter for slow responses (>5 seconds)
$ aws logs put-metric-filter \
    --log-group-name /ecs/fargate-app \
    --filter-name "SlowResponses" \
    --filter-pattern '[timestamp, requestId, duration > 5000, ...]' \
    --metric-transformations \
        metricName=SlowRequests,metricNamespace=Custom/App,metricValue=1,defaultValue=0

---

Step 4 — Create Auto-Remediation Lambda

Lambda Function — Restart Unhealthy Instance

auto-remediate/index.py:

import json
import boto3
import os

ec2 = boto3.client('ec2')
sns = boto3.client('sns')

TOPIC_ARN = os.environ['SNS_TOPIC_ARN']

def handler(event, context):
    print(f"Received event: {json.dumps(event)}")

    # Extract instance ID from CloudWatch Alarm
    detail = event.get('detail', {})
    instance_id = None

    # Handle EC2 state change events
    if event.get('source') == 'aws.ec2':
        instance_id = detail.get('instance-id')
        state = detail.get('state')

        if state == 'stopped':
            print(f"Instance {instance_id} stopped. Restarting...")
            ec2.start_instances(InstanceIds=[instance_id])

            sns.publish(
                TopicArn=TOPIC_ARN,
                Subject=f"Auto-Remediation: Instance {instance_id} Restarted",
                Message=json.dumps({
                    'action': 'restart',
                    'instance_id': instance_id,
                    'reason': 'Instance unexpectedly stopped',
                    'status': 'restarted'
                }, indent=2)
            )

    # Handle CloudWatch Alarm events
    elif event.get('source') == 'aws.cloudwatch':
        alarm_name = detail.get('alarmName', '')
        state_value = detail.get('state', {}).get('value', '')

        if state_value == 'ALARM' and 'HighCPU' in alarm_name:
            # Could trigger scaling or instance replacement
            sns.publish(
                TopicArn=TOPIC_ARN,
                Subject=f"Alert: {alarm_name} in ALARM state",
                Message=json.dumps({
                    'alarm': alarm_name,
                    'state': state_value,
                    'reason': detail.get('state', {}).get('reason', ''),
                    'action': 'notification_sent'
                }, indent=2)
            )

    return {'statusCode': 200, 'body': 'Processed'}

Deploy the Lambda

# Create the Lambda role
$ aws iam create-role \
    --role-name auto-remediation-role \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {"Service": "lambda.amazonaws.com"},
        "Action": "sts:AssumeRole"
      }]
    }'

$ aws iam put-role-policy \
    --role-name auto-remediation-role \
    --policy-name remediation-policy \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": ["ec2:StartInstances", "ec2:StopInstances", "ec2:DescribeInstances"],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": ["sns:Publish"],
          "Resource": "arn:aws:sns:us-east-1:123456789012:critical-alerts"
        },
        {
          "Effect": "Allow",
          "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
          "Resource": "*"
        }
      ]
    }'

# Package and deploy
$ cd auto-remediate && zip function.zip index.py && cd ..
$ aws lambda create-function \
    --function-name auto-remediation \
    --runtime python3.12 \
    --handler index.handler \
    --role arn:aws:iam::123456789012:role/auto-remediation-role \
    --zip-file fileb://auto-remediate/function.zip \
    --environment Variables="{SNS_TOPIC_ARN=arn:aws:sns:us-east-1:123456789012:critical-alerts}" \
    --timeout 30 \
    --tags Project=practice-5

---

Step 5 — Create EventBridge Rules

Rule 1: EC2 Instance State Changes

$ aws events put-rule \
    --name "ec2-state-change" \
    --description "Trigger on EC2 instance state changes" \
    --event-pattern '{
      "source": ["aws.ec2"],
      "detail-type": ["EC2 Instance State-change Notification"],
      "detail": {
        "state": ["stopped", "terminated"]
      }
    }' \
    --tags Key=Project,Value=practice-5

# Add Lambda as target
$ aws events put-targets \
    --rule ec2-state-change \
    --targets "Id=auto-remediation,Arn=arn:aws:lambda:us-east-1:123456789012:function:auto-remediation"

# Grant EventBridge permission to invoke Lambda
$ aws lambda add-permission \
    --function-name auto-remediation \
    --statement-id eventbridge-ec2 \
    --action lambda:InvokeFunction \
    --principal events.amazonaws.com \
    --source-arn arn:aws:events:us-east-1:123456789012:rule/ec2-state-change

Rule 2: CloudWatch Alarm State Changes

$ aws events put-rule \
    --name "alarm-state-change" \
    --description "Trigger on CloudWatch alarm state changes" \
    --event-pattern '{
      "source": ["aws.cloudwatch"],
      "detail-type": ["CloudWatch Alarm State Change"],
      "detail": {
        "state": {
          "value": ["ALARM"]
        }
      }
    }'

$ aws events put-targets \
    --rule alarm-state-change \
    --targets "Id=auto-remediation,Arn=arn:aws:lambda:us-east-1:123456789012:function:auto-remediation"

$ aws lambda add-permission \
    --function-name auto-remediation \
    --statement-id eventbridge-alarm \
    --action lambda:InvokeFunction \
    --principal events.amazonaws.com \
    --source-arn arn:aws:events:us-east-1:123456789012:rule/alarm-state-change

Rule 3: Scheduled Health Check (every 5 minutes)

$ aws events put-rule \
    --name "scheduled-health-check" \
    --description "Run health check every 5 minutes" \
    --schedule-expression "rate(5 minutes)"

$ aws events put-targets \
    --rule scheduled-health-check \
    --targets "Id=health-check,Arn=arn:aws:lambda:us-east-1:123456789012:function:auto-remediation"

---

Step 6 — Create CloudWatch Dashboard

$ aws cloudwatch put-dashboard \
    --dashboard-name "Production-Overview" \
    --dashboard-body '{
      "widgets": [
        {
          "type": "metric",
          "x": 0, "y": 0, "width": 12, "height": 6,
          "properties": {
            "title": "EC2 CPU Utilization",
            "metrics": [
              ["AWS/EC2", "CPUUtilization", "InstanceId", "i-0abc123", {"stat": "Average"}]
            ],
            "period": 300,
            "view": "timeSeries"
          }
        },
        {
          "type": "metric",
          "x": 12, "y": 0, "width": 12, "height": 6,
          "properties": {
            "title": "ALB Request Count",
            "metrics": [
              ["AWS/ApplicationELB", "RequestCount", "LoadBalancer", "app/prod-alb/abc123", {"stat": "Sum"}]
            ],
            "period": 60,
            "view": "timeSeries"
          }
        },
        {
          "type": "metric",
          "x": 0, "y": 6, "width": 12, "height": 6,
          "properties": {
            "title": "RDS Connections",
            "metrics": [
              ["AWS/RDS", "DatabaseConnections", "DBInstanceIdentifier", "prod-db", {"stat": "Average"}]
            ],
            "period": 300,
            "view": "timeSeries"
          }
        },
        {
          "type": "metric",
          "x": 12, "y": 6, "width": 12, "height": 6,
          "properties": {
            "title": "Application Errors",
            "metrics": [
              ["Custom/App", "ApplicationErrors", {"stat": "Sum"}]
            ],
            "period": 300,
            "view": "timeSeries"
          }
        },
        {
          "type": "alarm",
          "x": 0, "y": 12, "width": 24, "height": 3,
          "properties": {
            "title": "Alarm Status",
            "alarms": [
              "arn:aws:cloudwatch:us-east-1:123456789012:alarm:EC2-HighCPU-Warning",
              "arn:aws:cloudwatch:us-east-1:123456789012:alarm:EC2-HighCPU-Critical",
              "arn:aws:cloudwatch:us-east-1:123456789012:alarm:ALB-High5xxRate",
              "arn:aws:cloudwatch:us-east-1:123456789012:alarm:RDS-LowStorage"
            ]
          }
        }
      ]
    }'

Dashboard URL

https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=Production-Overview

---

Step 7 — Test the System

# Test 1: Trigger a CPU alarm (stress test)
$ aws ssm send-command \
    --document-name "AWS-RunShellScript" \
    --targets "Key=instanceids,Values=i-0abc123" \
    --parameters 'commands=["stress --cpu 4 --timeout 600"]'

# Monitor the alarm
$ watch -n 30 'aws cloudwatch describe-alarms \
    --alarm-names EC2-HighCPU-Warning \
    --query "MetricAlarms[0].StateValue"'

# Test 2: Stop an instance to trigger auto-remediation
$ aws ec2 stop-instances --instance-ids i-0abc123

# Check Lambda logs
$ aws logs tail /aws/lambda/auto-remediation --follow

# Test 3: Verify email notification arrived
# Check your inbox for SNS notification

---

Step 8 — Clean Up

# Delete EventBridge rules and targets
$ aws events remove-targets --rule ec2-state-change --ids auto-remediation
$ aws events delete-rule --name ec2-state-change
$ aws events remove-targets --rule alarm-state-change --ids auto-remediation
$ aws events delete-rule --name alarm-state-change
$ aws events remove-targets --rule scheduled-health-check --ids health-check
$ aws events delete-rule --name scheduled-health-check

# Delete Lambda function
$ aws lambda delete-function --function-name auto-remediation

# Delete CloudWatch alarms
$ aws cloudwatch delete-alarms --alarm-names \
    EC2-HighCPU-Warning EC2-HighCPU-Critical \
    ALB-High5xxRate RDS-LowStorage \
    Billing-Exceeds-50USD App-HighErrorRate

# Delete dashboard
$ aws cloudwatch delete-dashboards --dashboard-names Production-Overview

# Delete SNS topics
$ aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:123456789012:critical-alerts
$ aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:123456789012:warning-alerts
$ aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:123456789012:info-alerts

# Delete IAM role
$ aws iam delete-role-policy --role-name auto-remediation-role --policy-name remediation-policy
$ aws iam delete-role --role-name auto-remediation-role

---

What You Built

Component	Purpose
3 SNS Topics	Tiered notifications (Critical/Warning/Info)
6 CloudWatch Alarms	CPU, errors, storage, billing thresholds
2 Log Metric Filters	Application error and latency tracking
1 Lambda Function	Auto-remediation (restart, notify)
3 EventBridge Rules	Event-driven automation triggers
1 CloudWatch Dashboard	Unified monitoring view

---

Event Flow Summary

EC2 stops unexpectedly
     │
     ↓
EventBridge (ec2-state-change rule)
     │
     ↓
Lambda (auto-remediation)
     │
     ├── Restart the instance
     │
     └── Publish to SNS (critical-alerts)
              │
              ├── Email notification
              └── (Could add Slack, PagerDuty, etc.)

---

Common Mistakes

Mistake	Impact	Fix
Missing Lambda permissions	EventBridge can’t invoke Lambda	Add lambda:InvokeFunction resource policy
Wrong event pattern	Rules never trigger	Test with aws events test-event-pattern
Too sensitive thresholds	Alert fatigue	Tune evaluation periods and thresholds
No OK actions	Never know when issue resolves	Add --ok-actions to alarms
Single notification channel	Missed alerts	Use multiple: email + Slack + PagerDuty

---

Next → Practice Exercises