Amazon EKS — CloudWatch Container Insights

Container Insights is the AWS-native observability solution for EKS. It collects, aggregates, and visualizes metrics and logs at the cluster, node, pod, and container level — all within CloudWatch.

---

Enhanced Observability (Recommended)

EKS now offers enhanced observability as an EKS add-on:

aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability

# Verify
aws eks describe-addon --cluster-name my-cluster --addon-name amazon-cloudwatch-observability

This installs the CloudWatch agent and Fluent Bit as a DaemonSet, collecting:

• Cluster metrics (cluster, namespace, node, pod, container, service)
• Application logs
• Performance log events

---

Manual Installation (CloudWatch Agent)

If not using the add-on:

# Create namespace
kubectl create namespace amazon-cloudwatch

# Deploy CloudWatch agent
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonSet/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml

# Update cluster name and region
sed -i "s//my-cluster/g" cwagent-fluent-bit-quickstart.yaml
sed -i "s//us-east-1/g" cwagent-fluent-bit-quickstart.yaml

kubectl apply -f cwagent-fluent-bit-quickstart.yaml

IRSA for CloudWatch Agent

eksctl create iamserviceaccount \
  --cluster my-cluster \
  --namespace amazon-cloudwatch \
  --name cloudwatch-agent \
  --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
  --approve

---

Metrics Available

Level	Example Metrics
Cluster	node_count, failed_node_count
Namespace	pod_count, pod_cpu_utilization
Node	node_cpu_utilization, node_memory_utilization, node_filesystem_utilization
Pod	pod_cpu_utilization, pod_memory_utilization, pod_network_rx_bytes
Container	container_cpu_utilization, container_memory_utilization

---

CloudWatch Dashboards

Container Insights provides automatic dashboards in the CloudWatch console:

1. Go to CloudWatch → Container Insights
2. Select your cluster
3. View pre-built dashboards for:
   • Cluster overview
   • Node performance
   • Pod performance
   • EKS resource map (visual cluster topology)

Custom CloudWatch Alarms

aws cloudwatch put-metric-alarm \
  --alarm-name "EKS-High-CPU-Pod" \
  --metric-name pod_cpu_utilization \
  --namespace ContainerInsights \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --dimensions Name=ClusterName,Value=my-cluster Name=Namespace,Value=production \
  --alarm-actions arn:aws:sns:us-east-1:123456789:eks-alerts

---

CloudWatch Logs Insights Queries

Query Container Insights logs for debugging:

-- Find pods with OOM kills
fields @timestamp, kubernetes.pod_name, kubernetes.namespace_name
| filter @message like /OOMKilled/
| sort @timestamp desc

-- Top 10 pods by CPU usage
fields kubernetes.pod_name, pod_cpu_utilization
| stats avg(pod_cpu_utilization) as avg_cpu by kubernetes.pod_name
| sort avg_cpu desc
| limit 10

-- Container restart counts
fields kubernetes.pod_name, kubernetes.container_name
| filter kubernetes.container_restart_count > 0
| stats max(kubernetes.container_restart_count) as restarts by kubernetes.pod_name
| sort restarts desc

---

Fargate Logging

Fargate pods don’t run DaemonSets. Use the built-in Fluent Bit:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match *
        region us-east-1
        log_group_name /eks/my-cluster/fargate
        log_stream_prefix fargate-
        auto_create_group true

---

Cost Considerations

Component	Cost
CloudWatch metrics	$0.30 per metric/month (first 10,000)
Log ingestion	$0.50 per GB
Log storage	$0.03 per GB/month
Dashboard	Free (up to 3), $3/month (additional)
Container Insights enhanced	~$0.30 per metric + log costs

Cost tip: Container Insights can be expensive at scale. For large clusters, consider AMP + AMG or self-managed Prometheus.

---

Best Practices

1. Enable Container Insights add-on — simplest path for AWS-native observability
2. Use IRSA for CloudWatch agent — no node-level IAM for observability
3. Set log retention — don’t pay to store logs forever (aws logs put-retention-policy)
4. Create alarms for critical metrics — CPU, memory, pod restarts, node count
5. Use Logs Insights for ad-hoc debugging — faster than grep on raw logs

---