Amazon EKS — Right-Sizing & Resource Management

The average Kubernetes cluster wastes 30-60% of compute resources. Pods request far more CPU and memory than they use. Right-sizing eliminates this waste.

---

The Problem

Developers set resource requests conservatively (too high) because:

• They don’t know actual usage
• They’re afraid of OOMKills
• No tooling to recommend optimal values

Result: Nodes are “full” according to the scheduler, but 60% of CPU sits idle.

---

Analyzing Current Utilization

kubectl top

# Node utilization
kubectl top nodes

# Pod utilization
kubectl top pods --all-namespaces --sort-by=cpu

# Compare requests vs actual
kubectl top pods -n production --containers

Prometheus Queries

CPU request vs actual usage:

# CPU utilization ratio per namespace
sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m]))
/
sum(kube_pod_container_resource_requests{namespace="production", resource="cpu"})

Memory request vs actual:

# Memory utilization ratio
sum(container_memory_working_set_bytes{namespace="production"})
/
sum(kube_pod_container_resource_requests{namespace="production", resource="memory"})

---

VPA for Recommendations

VPA analyzes historical usage and recommends optimal resource requests:

# Install VPA
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vpa-v1-crd-gen.yaml
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vpa-rbac.yaml

# Or via Helm
helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa --namespace vpa --create-namespace

Create a VPA in recommendation-only mode (safe for production):

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"    # Recommendation only, no auto-update
  resourcePolicy:
    containerPolicies:
      - containerName: my-app
        minAllowed:
          cpu: 50m
          memory: 64Mi
        maxAllowed:
          cpu: 2000m
          memory: 4Gi

Check recommendations:

kubectl describe vpa my-app-vpa -n production

Output:

Recommendation:
  Container Recommendations:
    Container Name: my-app
    Lower Bound:
      Cpu:     25m
      Memory:  64Mi
    Target:
      Cpu:     100m      # Recommended request
      Memory:  256Mi     # Recommended request
    Upper Bound:
      Cpu:     500m
      Memory:  1Gi

---

Goldilocks Dashboard

Goldilocks runs VPA in recommendation mode for every Deployment and provides a web dashboard:

helm install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --create-namespace

# Enable for specific namespaces
kubectl label namespace production goldilocks.fairwinds.com/enabled=true

# Access dashboard
kubectl port-forward svc/goldilocks-dashboard -n goldilocks 8080:80

Goldilocks shows per-Deployment:

• Current requests
• VPA-recommended requests
• Potential savings (CPU and memory)

---

AWS Compute Optimizer

For node-level right-sizing (instance types):

# Enable Compute Optimizer
aws compute-optimizer update-enrollment-status --status Active

# Get recommendations for Auto Scaling groups
aws compute-optimizer get-auto-scaling-group-recommendations \
  --query "autoScalingGroupRecommendations[*].{Name:autoScalingGroupName,Current:currentConfiguration.instanceType,Recommended:recommendationOptions[0].configuration.instanceType}" \
  --output table

---

Resource Requests Best Practices

Resource	Request	Limit	Why
CPU	Set to P95 usage	Don’t set (or 2× request)	CPU is compressible; limits cause throttling
Memory	Set to P99 usage	Set = request (Guaranteed QoS)	Memory is incompressible; OOMKill if exceeded

resources:
  requests:
    cpu: 100m       # Based on VPA recommendation
    memory: 256Mi   # Based on VPA recommendation
  limits:
    # cpu: omitted — avoids unnecessary throttling
    memory: 256Mi   # Same as request = Guaranteed QoS for memory

---

Namespace-Level ResourceQuotas

Prevent teams from over-provisioning:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"

Combine with LimitRange for per-container defaults:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-a
spec:
  limits:
    - default:
        memory: 256Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      type: Container

---

Right-Sizing Node Groups

Current	Issue	Fix
All m6i.2xlarge nodes	Many pods use <1 vCPU	Mix smaller instance types
Single instance type	Can’t bin-pack efficiently	Diversify for better packing
Separate node group per team	Low utilization per group	Shared node group with namespaces

Use Karpenter — it automatically selects the optimal instance type per workload:

requirements:
  - key: node.kubernetes.io/instance-type
    operator: In
    values: ["m6i.large", "m6i.xlarge", "m6i.2xlarge", "c6i.large", "c6i.xlarge"]

---

Best Practices

1. Deploy VPA in Off mode — get recommendations without risk
2. Use Goldilocks — visual dashboard for all Deployments
3. Review quarterly — resource usage changes over time
4. Don’t set CPU limits — causes unnecessary throttling
5. Set memory limits = requests — predictable QoS behavior
6. Use ResourceQuotas per namespace — shared responsibility with dev teams

---