Amazon EKS — Self-Managed Nodes

Self-managed nodes give you full control over the EC2 instances running your EKS workloads. You create the Auto Scaling Group, configure the launch template, and manage the node lifecycle yourself.

---

When to Use Self-Managed Nodes

Scenario	Why Self-Managed
Custom AMI builds	Bake security agents, monitoring tools, or custom kernels into the AMI
Specific kernel modules	Load eBPF programs, custom network drivers, or kernel parameters
Windows nodes	Windows managed node groups have limitations
Legacy migration	Moving from pre-managed-nodegroup EKS clusters
Advanced bootstrap	Complex user data scripts or initialization requirements

If none of these apply, use managed node groups instead. They’re simpler and AWS handles lifecycle management.

---

Architecture

You manage:                     AWS manages:
┌─────────────────┐            ┌─────────────────┐
│ Launch Template  │            │ EKS Control Plane│
│ Auto Scaling Grp │           │ API Server       │
│ AMI Updates      │           │ etcd             │
│ Node Draining    │           │ Certificates     │
│ Security Patches │           │ OIDC Provider    │
│ Bootstrap Script │           └─────────────────┘
└─────────────────┘

---

Step-by-Step Setup

Step 1 — Create the Node IAM Role

# Create the trust policy
cat > node-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create the role
aws iam create-role \
  --role-name eks-self-managed-node-role \
  --assume-role-policy-document file://node-trust-policy.json

# Attach required policies
aws iam attach-role-policy --role-name eks-self-managed-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy --role-name eks-self-managed-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
aws iam attach-role-policy --role-name eks-self-managed-node-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

# Create instance profile
aws iam create-instance-profile --instance-profile-name eks-self-managed-profile
aws iam add-role-to-instance-profile \
  --instance-profile-name eks-self-managed-profile \
  --role-name eks-self-managed-node-role

Step 2 — Get the EKS-Optimized AMI

# Get latest EKS-optimized AMI for your region and K8s version
AMI_ID=$(aws ssm get-parameter \
  --name /aws/service/eks/optimized-ami/1.29/amazon-linux-2023/x86_64/standard/recommended/image_id \
  --query 'Parameter.Value' --output text)

echo "AMI: $AMI_ID"

Step 3 — Create a Launch Template

# Get cluster details
CLUSTER_NAME="my-cluster"
API_SERVER=$(aws eks describe-cluster --name $CLUSTER_NAME --query 'cluster.endpoint' --output text)
CA_DATA=$(aws eks describe-cluster --name $CLUSTER_NAME --query 'cluster.certificateAuthority.data' --output text)
CLUSTER_CIDR=$(aws eks describe-cluster --name $CLUSTER_NAME --query 'cluster.kubernetesNetworkConfig.serviceIpv4Cidr' --output text)

# Create user data script
cat > userdata.sh << EOF
#!/bin/bash
/etc/eks/bootstrap.sh $CLUSTER_NAME \
  --apiserver-endpoint $API_SERVER \
  --b64-cluster-ca $CA_DATA \
  --kubelet-extra-args '--node-labels=node-type=self-managed'
EOF

# Base64 encode
USERDATA=$(base64 -w 0 userdata.sh)

# Create launch template
aws ec2 create-launch-template \
  --launch-template-name eks-self-managed-lt \
  --launch-template-data '{
    "ImageId": "'$AMI_ID'",
    "InstanceType": "t3.large",
    "IamInstanceProfile": {"Name": "eks-self-managed-profile"},
    "UserData": "'$USERDATA'",
    "SecurityGroupIds": ["sg-node-security-group"],
    "BlockDeviceMappings": [{
      "DeviceName": "/dev/xvda",
      "Ebs": {"VolumeSize": 50, "VolumeType": "gp3"}
    }],
    "TagSpecifications": [{
      "ResourceType": "instance",
      "Tags": [
        {"Key": "kubernetes.io/cluster/'$CLUSTER_NAME'", "Value": "owned"},
        {"Key": "Name", "Value": "eks-self-managed-node"}
      ]
    }]
  }'

Step 4 — Create the Auto Scaling Group

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name eks-self-managed-asg \
  --launch-template LaunchTemplateName=eks-self-managed-lt,Version='$Latest' \
  --min-size 2 \
  --max-size 10 \
  --desired-capacity 3 \
  --vpc-zone-identifier "subnet-private-a,subnet-private-b" \
  --tags \
    Key=kubernetes.io/cluster/$CLUSTER_NAME,Value=owned,PropagateAtLaunch=true \
    Key=k8s.io/cluster-autoscaler/enabled,Value=true,PropagateAtLaunch=true \
    Key=k8s.io/cluster-autoscaler/$CLUSTER_NAME,Value=owned,PropagateAtLaunch=true

Step 5 — Map the Node Role in aws-auth

For the nodes to join the cluster, the node IAM role must be mapped:

kubectl edit configmap aws-auth -n kube-system

Add:

mapRoles:
  - rolearn: arn:aws:iam::123456789:role/eks-self-managed-node-role
    username: system:node:{{EC2PrivateDNSName}}
    groups:
      - system:bootstrappers
      - system:nodes

Or if using EKS access entries (recommended):

aws eks create-access-entry \
  --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789:role/eks-self-managed-node-role \
  --type EC2_LINUX

---

Bootstrap Script Deep Dive

The EKS-optimized AMI includes /etc/eks/bootstrap.sh. Key parameters:

Parameter	Description
--apiserver-endpoint	EKS API server URL
--b64-cluster-ca	Base64-encoded cluster CA certificate
--kubelet-extra-args	Additional kubelet flags (labels, taints, max-pods)
--container-runtime	Container runtime (containerd is default for 1.24+)
--dns-cluster-ip	DNS service IP (usually 10.100.0.10)

Example with Extra Args

/etc/eks/bootstrap.sh my-cluster \
  --kubelet-extra-args '--node-labels=workload=cpu-intensive --register-with-taints=dedicated=cpu:NoSchedule --max-pods=58'

---

Custom AMI Builds with Packer

For security-hardened or pre-configured nodes, build custom AMIs:

{
  "builders": [{
    "type": "amazon-ebs",
    "source_ami_filter": {
      "filters": {
        "name": "amazon-eks-node-1.29-*",
        "architecture": "x86_64"
      },
      "owners": ["602401143452"],
      "most_recent": true
    },
    "instance_type": "t3.large",
    "ssh_username": "ec2-user",
    "ami_name": "custom-eks-node-1.29-{{timestamp}}"
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sudo yum update -y",
      "sudo yum install -y amazon-cloudwatch-agent",
      "sudo systemctl enable amazon-cloudwatch-agent"
    ]
  }]
}

packer build custom-eks-node.json

---

Self-Managed vs Managed Node Groups

Feature	Self-Managed	Managed
AMI selection	Full control	AWS default or custom via launch template
Node draining	Manual (you script it)	Automatic during updates
Rolling updates	Manual (via ASG)	AWS handles it
Health replacement	ASG health checks only	EKS + ASG health checks
Setup complexity	High	Low
Node joining	aws-auth / access entry required	Automatic
Customization	Unlimited	Launch template limited

---

Upgrade Strategy for Self-Managed Nodes

Since AWS doesn’t manage upgrades, you need a strategy:

Rolling Replacement (In-Place)

1. Build new AMI with updated Kubernetes version
2. Update launch template with new AMI
3. Start instance refresh on ASG
4. ASG replaces nodes one at a time

aws autoscaling start-instance-refresh \
  --auto-scaling-group-name eks-self-managed-asg \
  --preferences MinHealthyPercentage=90,InstanceWarmup=300

Blue-Green Node Groups

1. Create a new ASG with the updated AMI
2. Cordon old nodes
3. Drain pods from old nodes
4. Delete old ASG

# Cordon all old nodes
kubectl get nodes -l node-type=self-managed -o name | xargs kubectl cordon

# Drain one by one
kubectl get nodes -l node-type=self-managed -o name | xargs -I {} kubectl drain {} --ignore-daemonsets --delete-emptydir-data

---

Best Practices

1. Use managed node groups unless you have a specific reason for self-managed
2. Automate AMI builds with Packer and a CI/CD pipeline
3. Tag ASGs properly for Cluster Autoscaler discovery
4. Use instance refresh for rolling updates instead of manual termination
5. Monitor bootstrap failures — check /var/log/cloud-init-output.log on nodes
6. Use EKS access entries instead of aws-auth for node role mapping

---