Amazon EKS — Custom Networking and Prefix Delegation

Running EKS at scale, you’ll hit IP exhaustion sooner or later. Custom networking and prefix delegation are two techniques to solve this problem.

---

The IP Problem

By default, the VPC CNI assigns one VPC IP per pod from the same subnet as the node. With a /24 subnet (251 IPs), you can quickly run out of IPs as your cluster grows.

Example: 10 nodes × 29 pods/node = 290 pod IPs + 10 node IPs = 300 IPs needed — more than a /24 subnet provides.

---

Solution 1 — Custom Networking (Secondary CIDR)

Custom networking lets you use a different CIDR range for pod IPs while keeping nodes in the original subnets.

How It Works

1. Add a secondary CIDR to your VPC (e.g., 100.64.0.0/16)
2. Create new subnets in the secondary CIDR
3. Configure the VPC CNI to assign pod IPs from these new subnets
4. Node IPs stay in the original subnets; pod IPs come from the secondary CIDR

VPC: 10.0.0.0/16 (primary) + 100.64.0.0/16 (secondary)

Nodes: 10.0.1.0/24, 10.0.2.0/24, 10.0.3.0/24
Pods:  100.64.0.0/19, 100.64.32.0/19, 100.64.64.0/19

Step-by-Step Setup

1. Add Secondary CIDR to VPC

aws ec2 associate-vpc-cidr-block \
  --vpc-id vpc-123456 \
  --cidr-block 100.64.0.0/16

2. Create Subnets in the Secondary CIDR

aws ec2 create-subnet \
  --vpc-id vpc-123456 \
  --cidr-block 100.64.0.0/19 \
  --availability-zone us-east-1a \
  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=eks-pod-subnet-a}]'

3. Enable Custom Networking on VPC CNI

kubectl set env daemonset aws-node -n kube-system \
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

4. Create ENIConfig Resources

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-east-1a
spec:
  subnet: subnet-abc123     # Secondary CIDR subnet in AZ-a
  securityGroups:
    - sg-xyz789              # Cluster security group
---
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-east-1b
spec:
  subnet: subnet-def456     # Secondary CIDR subnet in AZ-b
  securityGroups:
    - sg-xyz789

5. Set ENI Config Label Selector

kubectl set env daemonset aws-node -n kube-system \
  ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

Important: Existing pods won’t get new IPs. You need to restart workloads for pod IPs to come from the new subnets.

---

Solution 2 — Prefix Delegation

Prefix delegation assigns a /28 prefix (16 IPs) to each ENI slot instead of a single IP. This dramatically increases the number of pods per node.

Before vs After

Instance Type	Default Max Pods	With Prefix Delegation Max Pods
t3.medium	17	110
m5.large	29	110
m5.xlarge	58	110

With prefix delegation, most instances can run up to 110 pods (the EKS default max, adjustable with --max-pods).

How It Works

Instead of assigning one secondary IP per ENI slot, the VPC CNI requests a /28 prefix (16 contiguous IPs) per slot.

Without Prefix Delegation:
  ENI-0: IP, IP, IP, IP, IP, IP, IP, IP, IP  (9 slots = 9 IPs)

With Prefix Delegation:
  ENI-0: /28, /28, /28, /28, /28, /28, /28, /28, /28  (9 slots × 16 IPs = 144 IPs)

Enable Prefix Delegation

kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true

For new clusters, you can also set this in the EKS add-on configuration:

# Terraform
cluster_addons = {
  vpc-cni = {
    most_recent = true
    configuration_values = jsonencode({
      env = {
        ENABLE_PREFIX_DELEGATION = "true"
        WARM_PREFIX_TARGET       = "1"
      }
    })
  }
}

Increase Max Pods on Nodes

After enabling prefix delegation, you need to update the max pods setting on your nodes:

# For managed node groups, set via launch template user data
# AL2023 uses nodeadm
--kubelet-extra-args '--max-pods=110'

Requirements and Gotchas

Requirement	Details
VPC CNI version	v1.9.0+
Instance type	Nitro-based instances only
Subnet size	Needs /28 aligned contiguous blocks available
Existing pods	Must restart DaemonSet and workloads after enabling
IP planning	Each /28 block consumes 16 IPs from the subnet — need larger subnets

---

Custom Networking + Prefix Delegation

You can combine both techniques:

• Custom networking: Pods use a secondary CIDR (more IP space)
• Prefix delegation: Each ENI slot gets 16 IPs (more pods per node)

This gives you maximum scalability — thousands of pods with abundant IP space.

---

Which Should You Use?

Scenario	Recommendation
Small cluster, IPs not a concern	Default VPC CNI (no changes)
Running out of IPs in node subnets	Custom Networking (secondary CIDR)
Need more pods per node	Prefix Delegation
Large cluster, both problems	Custom Networking + Prefix Delegation

---

What’s Next?

Let’s learn about applying VPC security groups directly to pods.

Next: Security Groups for Pods →