Amazon EKS — Karpenter

Karpenter is AWS’s next-generation node provisioner. Unlike Cluster Autoscaler (which works through ASGs), Karpenter provisions EC2 instances directly — choosing the right instance type, size, and purchase option for each workload in real time.

---

Why Karpenter over Cluster Autoscaler?

Feature	Cluster Autoscaler	Karpenter
Scales via	ASG desired count	EC2 directly (RunInstances)
Instance type selection	Fixed per ASG	Dynamic — picks optimal per workload
Scale-up speed	2-5 minutes	30-90 seconds
Bin-packing	No	Yes — consolidation and right-sizing
Spot handling	ASG mixed instances	Native diversification
Node lifecycle	ASG manages	Karpenter manages directly
Configuration	Per-ASG tags + CA flags	NodePool + EC2NodeClass CRDs

---

Installation

Prerequisites

export KARPENTER_VERSION="1.0.0"
export CLUSTER_NAME="my-cluster"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

Helm Install

helm registry logout public.ecr.aws || true

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace kube-system \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

---

NodePool

NodePool defines what Karpenter can provision:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        team: platform
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["medium", "large", "xlarge", "2xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h    # Recycle nodes every 30 days
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
  limits:
    cpu: "100"
    memory: 400Gi
  weight: 50

---

EC2NodeClass

EC2NodeClass defines AWS-specific configuration:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-my-cluster
  amiSelectorTerms:
    - alias: al2023@latest    # Amazon Linux 2023
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 3000
        throughput: 125
        encrypted: true
  tags:
    Environment: production
    ManagedBy: karpenter

---

Consolidation

Karpenter actively optimizes node utilization:

• WhenEmpty: Remove nodes with no running pods
• WhenEmptyOrUnderutilized: Replace nodes when workloads can fit on fewer/smaller nodes

Example consolidation flow:

1. You have 3x m5.xlarge nodes at 30% utilization
2. Karpenter calculates workloads fit on 2x m5.large
3. It drains one node, then replaces the others with smaller instances
4. Result: fewer nodes, right-sized, lower cost

---

Spot Instance Handling

spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]

Karpenter automatically:

• Diversifies across instance types for Spot
• Handles Spot interruption notices
• Replaces interrupted Spot nodes with new ones

Separate Pools for Critical vs Non-Critical

# Critical workloads — On-Demand only
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: critical
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
      taints:
        - key: workload-type
          value: critical
          effect: NoSchedule
---
# Non-critical — Spot preferred
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: batch
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]

---

Migration from Cluster Autoscaler

1. Install Karpenter alongside Cluster Autoscaler
2. Create NodePool and EC2NodeClass
3. Taint existing ASG nodes: kubectl taint nodes -l node-group=old CAStoKarpenter=migrating:NoSchedule
4. New pods schedule on Karpenter nodes
5. Drain old nodes: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
6. Scale down ASGs to 0
7. Remove Cluster Autoscaler

---

Best Practices

1. Use Karpenter for all new clusters — faster, smarter, more cost-effective
2. Enable consolidation — WhenEmptyOrUnderutilized for best cost savings
3. Set limits — prevent runaway spending with CPU/memory caps
4. Use expireAfter — recycle nodes for patching and drift prevention
5. Diversify instance types — allow multiple families (c, m, r) and sizes
6. Use Spot for non-critical workloads — Karpenter handles interruptions automatically

---