AWS for DevOps — Monitoring Exercises

3 minute read

10.6 — Practice Exercises: Monitoring & Observability

Test your understanding of CloudWatch, CloudTrail, X-Ray, and SNS.

Exercise 1 — Create a CloudWatch Alarm

Create a CloudWatch alarm that triggers when an EC2 instance’s CPU exceeds 80% for 2 consecutive 5-minute periods. The alarm should send a notification to an SNS topic called infra-alerts.

Show Answer 
# Create the SNS topic
$ aws sns create-topic --name infra-alerts

# Subscribe your email
$ aws sns subscribe \
    --topic-arn arn:aws:sns:ap-south-1:123456789012:infra-alerts \
    --protocol email \
    --notification-endpoint admin@example.com

# Create the alarm
$ aws cloudwatch put-metric-alarm \
    --alarm-name high-cpu-ec2 \
    --metric-name CPUUtilization \
    --namespace AWS/EC2 \
    --dimensions Name=InstanceId,Value=i-0abc123def456 \
    --statistic Average \
    --period 300 \
    --threshold 80 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 2 \
    --alarm-actions arn:aws:sns:ap-south-1:123456789012:infra-alerts \
    --ok-actions arn:aws:sns:ap-south-1:123456789012:infra-alerts

Key points:

  • --period 300 = 5-minute intervals
  • --evaluation-periods 2 = must breach for 2 consecutive periods
  • --ok-actions notifies when the alarm returns to OK state

Exercise 2 — Query CloudWatch Logs

You have a log group /ecs/api and need to find all requests that took longer than 2 seconds in the last hour. Write the Logs Insights query.

Show Answer 
$ aws logs start-query \
    --log-group-name /ecs/api \
    --start-time $(date -u -v-1H +%s) \
    --end-time $(date -u +%s) \
    --query-string '
        fields @timestamp, @message
        | filter @message like /response_time/
        | parse @message "response_time=*ms" as rt
        | filter rt > 2000
        | sort @timestamp desc
        | limit 50
    '

Alternative if using JSON logs:

fields @timestamp, request_path, response_time
| filter response_time > 2000
| sort response_time desc
| limit 50

Key points:

  • parse extracts fields from unstructured log messages
  • filter narrows results to slow requests
  • Use stats avg(response_time) by request_path to find which endpoints are slow

Exercise 3 — Investigate with CloudTrail

Someone deleted a production S3 bucket last night. How would you find out who did it and when?

Show Answer 
# Look up DeleteBucket events in the last 24 hours
$ aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket \
    --start-time "2026-03-29T00:00:00Z" \
    --end-time "2026-03-30T12:00:00Z"

The output will show:

  • Username — who made the API call
  • EventTime — when it happened
  • SourceIPAddress — from where
  • Resources — which bucket was deleted

To prevent this in the future:

  1. Enable MFA Delete on critical buckets
  2. Use S3 Bucket Policy to deny s3:DeleteBucket
  3. Set up a CloudWatch alarm on DeleteBucket events via CloudTrail + CloudWatch Logs

Exercise 4 — Design a Monitoring Strategy

Your team runs a microservices application with: API Gateway → Lambda → DynamoDB. Design a monitoring strategy covering metrics, logs, traces, and alerts.

Show Answer

Metrics to monitor:

Service Metric Threshold
API Gateway 5XXError > 5 per minute
API Gateway Latency (p99) > 3 seconds
Lambda Errors > 0
Lambda Duration (p99) > 80% of timeout
Lambda Throttles > 0
DynamoDB ConsumedReadCapacityUnits > 80% of provisioned
DynamoDB SystemErrors > 0

Logging strategy:

  • API Gateway: Enable access logging (JSON format)
  • Lambda: Structured JSON logs with correlation ID
  • Use a shared log group prefix: /app/my-api/

Tracing:

  • Enable X-Ray Active tracing on both API Gateway and Lambda
  • Use X-Ray SDK in Lambda to trace DynamoDB calls
  • Set up service map dashboard

Alert routing:

Condition Severity Action
Lambda errors > 0 for 5 min P2 Slack #incidents
API Gateway 5XX > 10 for 5 min P1 PagerDuty
DynamoDB throttles > 0 P3 Slack #monitoring
Lambda near timeout P3 Slack #monitoring

Exercise 5 — Set Up CloudTrail + CloudWatch Integration

Create a CloudTrail trail that sends management events to a CloudWatch Logs group. Then create a metric filter that counts DeleteBucket events and an alarm that fires when any bucket is deleted.

Show Answer 
# 1. Create the S3 bucket for trail logs
$ aws s3 mb s3://my-trail-logs-123456

# 2. Create the CloudWatch log group
$ aws logs create-log-group --log-group-name /aws/cloudtrail/audit

# 3. Create the trail with CloudWatch integration
$ aws cloudtrail create-trail \
    --name audit-trail \
    --s3-bucket-name my-trail-logs-123456 \
    --is-multi-region-trail \
    --enable-log-file-validation \
    --cloud-watch-logs-log-group-arn \
      arn:aws:logs:ap-south-1:123456789012:log-group:/aws/cloudtrail/audit:* \
    --cloud-watch-logs-role-arn \
      arn:aws:iam::123456789012:role/CloudTrailToCloudWatch

# 4. Start logging
$ aws cloudtrail start-logging --name audit-trail

# 5. Create a metric filter for DeleteBucket
$ aws logs put-metric-filter \
    --log-group-name /aws/cloudtrail/audit \
    --filter-name delete-bucket-events \
    --filter-pattern '{ $.eventName = "DeleteBucket" }' \
    --metric-transformations \
      metricName=DeleteBucketCount,metricNamespace=SecurityAlerts,metricValue=1

# 6. Create an alarm on the metric
$ aws cloudwatch put-metric-alarm \
    --alarm-name bucket-deletion-alert \
    --metric-name DeleteBucketCount \
    --namespace SecurityAlerts \
    --statistic Sum \
    --period 300 \
    --threshold 0 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 1 \
    --alarm-actions arn:aws:sns:ap-south-1:123456789012:critical-alerts

Key points:

  • The metric filter scans CloudTrail logs in real-time via CloudWatch Logs
  • Any DeleteBucket event increments the custom metric
  • The alarm fires immediately (1 evaluation period) when any bucket is deleted

Next Chapter → Configuration & Secrets Management

Updated:

You may also enjoy