Amazon EKS — Right-Sizing Workloads

Most EKS clusters are over-provisioned. Developers set high resource requests to be safe, and nobody revisits them. Right-sizing means setting resource requests and limits to match actual usage — saving money and improving scheduling efficiency.

---

The Problem

Common Mistake	Impact
Requests too high	Nodes fill up faster, need more nodes than necessary
Requests too low	Pods get OOMKilled or CPU-throttled
No limits set	Noisy neighbor problem — one pod can starve others
Limits = Requests	No burst capacity, guaranteed QoS (sometimes too strict)

---

Analyzing Resource Utilization

kubectl top

# Current pod resource usage
kubectl top pods -n production --sort-by=cpu
kubectl top pods -n production --sort-by=memory

# Current node usage
kubectl top nodes

Prometheus Queries

# CPU request vs actual usage by deployment
sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m])) by (pod)
/
sum(kube_pod_container_resource_requests{resource="cpu", namespace="production"}) by (pod)

# Memory request vs actual usage
sum(container_memory_working_set_bytes{namespace="production"}) by (pod)
/
sum(kube_pod_container_resource_requests{resource="memory", namespace="production"}) by (pod)

---

VPA Recommendations

Run VPA in Off mode to get recommendations without automatic changes:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-app-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  updatePolicy:
    updateMode: "Off"

kubectl describe vpa web-app-vpa -n production

---

Goldilocks

Goldilocks provides a dashboard of VPA recommendations for all deployments:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks --create-namespace

# Enable VPA for a namespace
kubectl label namespace production goldilocks.fairwinds.com/enabled=true

Access the dashboard:

kubectl port-forward svc/goldilocks-dashboard 8080:80 -n goldilocks

Open http://localhost:8080 to see recommendations per deployment.

---

AWS Compute Optimizer

Compute Optimizer analyzes node-level utilization and recommends instance types:

aws compute-optimizer get-ec2-instance-recommendations \
  --filters name=Finding,values=OVER_PROVISIONED \
  --query 'instanceRecommendations[*].{Instance:instanceArn,Finding:finding,Current:currentInstanceType,Recommended:recommendationOptions[0].instanceType}'

---

Resource Request Best Practices

Workload Type	CPU Request	Memory Request	CPU Limit	Memory Limit
Web server	P99 usage	P99 + 20% headroom	2-3x request	Request + 30%
Background worker	Average usage	P99 usage	2-3x request	Request + 30%
Batch job	Average usage	Peak usage	None (burstable)	Peak + 20%
Database	Steady-state usage	Allocated + cache	Equal to request	Equal to request

Example

If a web app uses 150m CPU at P99 and 256Mi memory at P99:

resources:
  requests:
    cpu: 150m
    memory: 300Mi     # 256Mi + 20% headroom
  limits:
    cpu: 400m         # ~2.5x request for burst
    memory: 400Mi     # 300Mi + 30%

---

QoS Classes

Kubernetes assigns QoS classes based on resource configuration:

QoS Class	Condition	Eviction Priority
Guaranteed	requests == limits (both CPU and memory)	Last to be evicted
Burstable	requests < limits (or only requests set)	Middle priority
BestEffort	No requests or limits	First to be evicted

Production workloads should be Guaranteed or Burstable — never BestEffort.

---

Namespace ResourceQuotas

Prevent teams from over-requesting:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "100"

LimitRange (Default per Pod)

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
    - default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      max:
        cpu: 4
        memory: 8Gi
      min:
        cpu: 50m
        memory: 64Mi
      type: Container

---

Best Practices

1. Start with VPA recommendations — don’t guess resource requests
2. Use Goldilocks for cluster-wide visibility
3. Set both requests and limits — requests for scheduling, limits for protection
4. Review resource usage monthly — workloads change over time
5. Use ResourceQuotas per namespace — prevent resource hoarding
6. Right-size nodes too — AWS Compute Optimizer helps choose instance types

---