Amazon EKS — Custom Networking & Prefix Delegation

1 minute read

Custom Networking & Prefix Delegation

When standard VPC CNI networking isn’t enough — you’re running out of IPs, need pods in different subnets, or need more pods per node — custom networking and prefix delegation help.

Prefix Delegation

Prefix delegation assigns a /28 prefix (16 IPs) to each ENI slot instead of individual IPs. This dramatically increases pods per node:

Without Prefix Delegation:          With Prefix Delegation:
┌──────────────────┐                ┌──────────────────┐
│ ENI slot → 1 IP  │                │ ENI slot → /28   │
│                  │                │ (16 IPs)         │
│ m5.large:        │                │ m5.large:        │
│ 3 ENIs × 10 IPs │                │ 3 ENIs × 9 slots │
│ = 29 max pods    │                │ × 16 = 432 IPs   │
│                  │                │ = 110 max pods    │
└──────────────────┘                └──────────────────┘

Enable Prefix Delegation

# Set environment variable on aws-node DaemonSet
$ kubectl set env daemonset aws-node -n kube-system \
    ENABLE_PREFIX_DELEGATION=true

# Increase max pods per node
$ kubectl set env daemonset aws-node -n kube-system \
    WARM_PREFIX_TARGET=1

With eksctl:

managedNodeGroups:
  - name: general
    instanceType: m5.large
    maxPodsPerNode: 110      # Increase from default 29

Custom Networking

Custom networking puts pod IPs in different subnets than node IPs:

Custom Networking
┌──────────────────────────────────────────┐
│  Node subnet: 10.0.0.0/24               │
│  Node IP: 10.0.0.10                      │
│                                          │
│  Pod subnet: 100.64.0.0/16              │
│  Pod IPs: 100.64.1.10, 100.64.1.11, ... │
│                                          │
│  → Pods use secondary CIDR               │
│  → Saves primary VPC IPs for nodes       │
└──────────────────────────────────────────┘

Enable:

$ kubectl set env daemonset aws-node -n kube-system \
    AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

Then create ENIConfig resources for each AZ:

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-east-1a
spec:
  subnet: subnet-0abc123     # Pod subnet in AZ-a
  securityGroups:
    - sg-0def456

When to Use Each

Feature Use When
Standard VPC CNI Default — works for most clusters
Prefix delegation Need more pods per node (>29 on m5.large)
Custom networking Running out of primary VPC IPs
Secondary CIDRs VPC CIDR too small for pod IPs

Common Mistakes

Mistake Impact
Prefix delegation without increasing max pods Still limited to default
Custom networking without ENIConfig per AZ Pods can’t get IPs
Forgetting to add secondary CIDR to VPC No IPs for pods
Not testing with existing workloads Potential connectivity issues

What’s Next?

Test your knowledge: Networking Exercises.

You may also enjoy