Amazon EKS — Managed Node Groups

Managed node groups are the recommended way to run EC2 worker nodes on EKS. AWS handles AMI updates, rolling upgrades, node draining, and lifecycle management — you just define the configuration.

---

What AWS Manages For You

Responsibility	Managed by AWS
EC2 instance provisioning	Yes — creates instances via Auto Scaling Group
EKS-optimized AMI	Yes — uses the latest EKS AMI by default
Node registration	Yes — bootstrap script joins nodes to the cluster
Rolling updates	Yes — cordons, drains, and replaces nodes during upgrades
Health monitoring	Yes — replaces unhealthy nodes automatically
Security patches	Partially — new AMI versions are released; you trigger the update

What you manage: instance types, scaling config, labels, taints, and when to trigger AMI updates.

---

Creating a Managed Node Group

Using eksctl

eksctl create nodegroup \
  --cluster my-cluster \
  --name app-nodes \
  --node-type t3.large \
  --nodes 3 \
  --nodes-min 2 \
  --nodes-max 10 \
  --managed \
  --asg-access

Using eksctl with ClusterConfig

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-east-1

managedNodeGroups:
  - name: app-nodes
    instanceType: t3.large
    desiredCapacity: 3
    minSize: 2
    maxSize: 10
    labels:
      role: application
    tags:
      Environment: production
    privateNetworking: true
    iam:
      withAddonPolicies:
        autoScaler: true
        cloudWatch: true

Using AWS CLI

aws eks create-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name app-nodes \
  --node-role arn:aws:iam::123456789:role/eks-node-role \
  --subnets subnet-private-a subnet-private-b \
  --instance-types t3.large \
  --scaling-config minSize=2,maxSize=10,desiredSize=3

Using Terraform

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "my-cluster"
  cluster_version = "1.29"

  eks_managed_node_groups = {
    app_nodes = {
      name           = "app-nodes"
      instance_types = ["t3.large"]
      min_size       = 2
      max_size       = 10
      desired_size   = 3

      labels = {
        role = "application"
      }

      tags = {
        Environment = "production"
      }
    }
  }
}

---

Launch Template Customization

You can customize managed node groups using a launch template for deeper control:

managedNodeGroups:
  - name: custom-nodes
    instanceType: m6i.xlarge
    desiredCapacity: 3
    launchTemplate:
      id: lt-0abc123def456789
      version: "1"

What You Can Customize with Launch Templates

Setting	Supported
Instance type	Yes
AMI	Yes (custom AMI)
User data (bootstrap)	Yes
Block device mappings	Yes
Security groups	Yes
Instance metadata options	Yes
Key pair (SSH)	Yes

Custom User Data Example

#!/bin/bash
# Additional bootstrap configuration
/etc/eks/bootstrap.sh my-cluster \
  --kubelet-extra-args '--max-pods=58 --node-labels=workload=high-performance'

---

Update Strategies

When you update a managed node group (new AMI, instance type change), EKS performs a rolling update:

Rolling Update Configuration

aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name app-nodes \
  --update-config maxUnavailable=1

Parameter	Description	Default
maxUnavailable	Maximum nodes updated simultaneously (absolute number)	1
maxUnavailablePercentage	Maximum nodes updated simultaneously (percentage)	—

Update Process (What AWS Does)

1. Launches new nodes with the updated configuration
2. Cordons old nodes (prevents new pod scheduling)
3. Drains pods from old nodes (respects PodDisruptionBudgets)
4. Terminates old nodes after pods are safely moved
5. Repeats until all nodes are updated

Force Update

If pods refuse to drain (e.g., no PDB, stuck pods):

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name app-nodes \
  --force

Warning: Force update terminates nodes without waiting for pods to drain gracefully.

---

Instance Types and Capacity Types

Choosing Instance Types

managedNodeGroups:
  - name: general-workloads
    instanceTypes:
      - m6i.xlarge      # Primary
      - m6i.2xlarge     # Fallback
      - m5.xlarge       # Fallback
    capacityType: ON_DEMAND

Multiple instance types increase availability when supply is limited (especially important for Spot).

Capacity Types

Type	Description	Use Case
ON_DEMAND	Standard pricing, guaranteed capacity	Production, stateful workloads
SPOT	Up to 90% cheaper, can be interrupted	Batch, dev/test, stateless workloads

---

Labels, Taints, and Tags

Apply Labels at Node Group Level

managedNodeGroups:
  - name: gpu-nodes
    labels:
      workload-type: ml-training
      accelerator: nvidia

Labels help with pod scheduling using nodeSelector or nodeAffinity.

Apply Taints

managedNodeGroups:
  - name: gpu-nodes
    taints:
      - key: nvidia.com/gpu
        value: "true"
        effect: NoSchedule

Taints prevent non-GPU workloads from scheduling on expensive GPU nodes.

AWS Tags

managedNodeGroups:
  - name: app-nodes
    tags:
      Team: platform
      CostCenter: "CC-1234"
      Environment: production

Tags flow through to EC2 instances and are useful for cost allocation.

---

Scaling Configuration

managedNodeGroups:
  - name: app-nodes
    minSize: 2      # Never go below 2 nodes
    maxSize: 20     # Allow up to 20 nodes
    desiredCapacity: 5  # Start with 5 nodes

Managed node groups use Auto Scaling Groups under the hood. Cluster Autoscaler or Karpenter adjusts the desired count based on pod demand.

---

Managed Node Group Lifecycle

CREATE → ACTIVE → UPDATING → ACTIVE → DEGRADED/DELETE_FAILED
                                              │
                                              └→ DELETE

Status	Meaning
CREATING	Node group is being created
ACTIVE	Healthy and running
UPDATING	Rolling update in progress
DEGRADED	Some nodes unhealthy
DELETE_FAILED	Cleanup failed (manual intervention needed)
DELETING	Being deleted

Check status:

aws eks describe-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name app-nodes \
  --query 'nodegroup.status'

---

Best Practices

1. Use private subnets for node groups — nodes don’t need public IPs
2. Set resource tags for cost allocation from day one
3. Use multiple instance types to increase availability
4. Configure PodDisruptionBudgets before enabling rolling updates
5. Don’t set desiredCapacity in Terraform if using Cluster Autoscaler — they’ll conflict
6. Use separate node groups for different workload types (general, GPU, high-memory)
7. Enable update notifications to know when new AMI versions are available

---