Amazon EKS — Security Groups for Pods

By default, all pods on a node share the node’s security groups. Security Groups for Pods lets you assign VPC security groups directly to individual pods — giving you fine-grained network isolation at the pod level.

---

Why Pod-Level Security Groups?

Without this feature, every pod on a node can access any resource the node’s security group allows. That’s a problem when:

• One pod needs RDS access but others shouldn’t have it
• Compliance requires pod-level network isolation
• You want to restrict which pods can reach specific AWS services

Security Groups for Pods solves this by attaching VPC security groups directly to pods using branch ENIs (Elastic Network Interfaces).

---

How It Works — Trunk and Branch ENIs

┌─────────────── EC2 Instance ──────────────────┐
│                                                │
│  Primary ENI (eth0)     ← Node networking      │
│    └─ SG: node-sg                              │
│                                                │
│  Trunk ENI              ← Created by VPC CNI   │
│    ├─ Branch ENI 1      ← Pod A                │
│    │    └─ SG: rds-access-sg                   │
│    ├─ Branch ENI 2      ← Pod B                │
│    │    └─ SG: redis-access-sg                 │
│    └─ Branch ENI 3      ← Pod C                │
│         └─ SG: default-pod-sg                  │
│                                                │
└────────────────────────────────────────────────┘

1. The VPC CNI creates a trunk ENI on each node
2. For pods needing custom security groups, VPC CNI creates branch ENIs off the trunk
3. Each branch ENI gets its own VPC security group(s)
4. The pod’s traffic flows through the branch ENI — enforced by VPC networking

---

Supported Instance Types

Not all EC2 instance types support trunk ENIs. You need Nitro-based instances:

Family	Supported	Examples
General Purpose	Yes	m5, m6i, m7i, m6g, m7g
Compute Optimized	Yes	c5, c6i, c7i, c6g, c7g
Memory Optimized	Yes	r5, r6i, r7i, r6g, r7g
T-series	Partial	t3 (yes), t2 (no)
Graviton	Yes	All Graviton instances

Fargate does not support Security Groups for Pods (Fargate pods get their own ENIs already).

Check the branch ENI limits per instance type — each type has a maximum number of branch ENIs.

---

Step-by-Step Setup

Step 1 — Enable Trunk ENI on VPC CNI

kubectl set env daemonset aws-node -n kube-system \
  ENABLE_POD_ENI=true

Verify:

kubectl get daemonset aws-node -n kube-system \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | jq '.[] | select(.name=="ENABLE_POD_ENI")'

Step 2 — Label Nodes for Trunk ENI

Nodes must have the vpc.amazonaws.com/has-trunk-attached label. VPC CNI adds this automatically after restarting the aws-node pods:

kubectl get nodes -o custom-columns="NAME:.metadata.name,TRUNK:.metadata.labels.vpc\.amazonaws\.com/has-trunk-attached"

Expected output:

NAME                                    TRUNK
ip-10-0-1-100.ec2.internal             true
ip-10-0-2-200.ec2.internal             true

Step 3 — Create a VPC Security Group

Create a security group that allows the pod to access RDS:

# Create the pod security group
aws ec2 create-security-group \
  --group-name pod-rds-access \
  --description "Allow pods to access RDS" \
  --vpc-id vpc-123456

# Allow inbound from this SG on the RDS security group
aws ec2 authorize-security-group-ingress \
  --group-id sg-rds-group-id \
  --protocol tcp \
  --port 5432 \
  --source-group sg-pod-rds-access-id

Step 4 — Create a SecurityGroupPolicy

The SecurityGroupPolicy CRD tells EKS which pods get which security groups:

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: rds-access-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      role: backend
      db-access: "true"
  securityGroups:
    groupIds:
      - sg-pod-rds-access-id    # Pod-specific security group
      - sg-cluster-default-id   # Cluster security group (required)

Important: Always include the cluster security group in the list. Without it, the pod can’t communicate with the API server or CoreDNS.

Step 5 — Deploy a Pod with the Matching Labels

apiVersion: v1
kind: Pod
metadata:
  name: backend-api
  namespace: production
  labels:
    role: backend
    db-access: "true"
spec:
  containers:
  - name: api
    image: your-ecr-repo/backend-api:latest
    ports:
    - containerPort: 8080

Step 6 — Verify

Check that the pod was scheduled with a branch ENI:

kubectl describe pod backend-api -n production | grep -i "vpc.amazonaws.com/pod-eni"

Expected annotation:

vpc.amazonaws.com/pod-eni: [{"eniId":"eni-abc123","ifAddress":"02:ab:cd:ef:12:34","privateIp":"10.0.1.55","vlanId":1,"subnetCidr":"10.0.1.0/24"}]

---

SecurityGroupPolicy Selectors

You can match pods using pod labels or service account names:

Match by Pod Labels

spec:
  podSelector:
    matchLabels:
      role: backend

Match by Service Account

spec:
  serviceAccountSelector:
    matchLabels:
      db-access: "true"

This is useful when you want to tie security group assignment to the pod’s identity rather than its workload labels.

---

Limitations and Gotchas

Limitation	Details
Instance types	Only Nitro-based instances support trunk ENIs
Branch ENI limits	Each instance type has a maximum number of branch ENIs
No Fargate	Fargate pods can’t use SecurityGroupPolicy
Network policy conflict	VPC CNI network policies and security groups for pods can interact unexpectedly
Performance	Branch ENI allocation adds a small delay to pod scheduling
Cluster SG required	Must include the cluster security group or pods lose API server/DNS access
Source NAT	Pods with branch ENIs bypass SNAT — their traffic uses the pod’s VPC IP directly

---

Common Use Cases

1 — Database Access Isolation

Only backend pods can reach RDS:

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: rds-only-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      tier: backend
  securityGroups:
    groupIds:
      - sg-allow-rds-5432
      - sg-cluster-sg

2 — Compliance Isolation

Isolate PCI workloads with a dedicated security group:

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: pci-workloads
  namespace: pci
spec:
  podSelector:
    matchLabels:
      compliance: pci-dss
  securityGroups:
    groupIds:
      - sg-pci-isolated
      - sg-cluster-sg

3 — Multi-Database Access

Some pods need access to both PostgreSQL and Redis:

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: multi-db-access
  namespace: production
spec:
  podSelector:
    matchLabels:
      role: data-processor
  securityGroups:
    groupIds:
      - sg-allow-postgres
      - sg-allow-redis
      - sg-cluster-sg

---

Troubleshooting

Pod stuck in Pending:

kubectl describe pod <pod-name> | grep -i "eni\|security\|branch"

If the node has run out of branch ENIs, the pod can’t be scheduled. Scale out or upgrade to an instance type with higher limits.

Pod can’t reach DNS:

You forgot to include the cluster security group. Update the SecurityGroupPolicy to include it.

Branch ENI not attaching:

Check VPC CNI version supports this feature (v1.11+):

kubectl describe daemonset aws-node -n kube-system | grep Image

---

Best Practices

1. Always include the cluster security group in every SecurityGroupPolicy
2. Use Nitro instances — t3.medium or larger for dev, m6i/m7i for production
3. Monitor branch ENI usage — watch for exhaustion under high pod density
4. Combine with IRSA or Pod Identity for complete pod-level isolation (network + IAM)
5. Use service account selectors when security group assignment should follow identity
6. Test in a non-production namespace first before rolling out broadly

---