CKS — Network Policies Advanced Lab

13 minute read

Lab Overview

This lab ties together all the advanced NetworkPolicy concepts from this chapter. You will:

  1. Create default deny policies (ingress + egress)
  2. Add a DNS egress allow policy
  3. Verify DNS works and external access is blocked
  4. Implement namespace isolation (frontend → backend, deny frontend → database)
  5. Block cloud metadata (169.254.169.254)
  6. Test all policies with temporary pods
  7. Debug an intentionally broken NetworkPolicy

Prerequisites

  • A Kubernetes cluster with a NetworkPolicy-supporting CNI (Calico or Cilium)
  • kubectl configured and working
  • Cluster-admin permissions

Cluster Setup

If you are using a local cluster (kind, minikube), ensure Calico is installed:

# For kind — create a cluster without default CNI and install Calico
kind create cluster --config=- <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: 192.168.0.0/16
EOF

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml

# Wait for Calico to be ready
kubectl wait --for=condition=Ready pods -l k8s-app=calico-node -n kube-system --timeout=120s

Exercise 1: Default Deny Ingress + Egress

Task

Create a namespace called lab-secure and apply default deny for both ingress and egress.

Steps

# Create the namespace
kubectl create namespace lab-secure

# Label it (we'll need this later)
kubectl label namespace lab-secure env=lab-secure

Create a file default-deny-all.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: lab-secure
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Apply it:

kubectl apply -f default-deny-all.yaml

Verify

kubectl get networkpolicy -n lab-secure

# Expected output:
# NAME               POD-SELECTOR   AGE
# default-deny-all   <none>         5s

Exercise 2: Allow DNS Egress

Task

Create a policy that allows all pods in lab-secure to reach CoreDNS on port 53.

Steps

Create a file allow-dns.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: lab-secure
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Apply it:

kubectl apply -f allow-dns.yaml

Verify

kubectl get networkpolicy -n lab-secure

# Expected output:
# NAME               POD-SELECTOR   AGE
# allow-dns          <none>         5s
# default-deny-all   <none>         2m

Exercise 3: Test DNS and External Access

Task

Verify that pods in lab-secure can resolve DNS but cannot reach external IPs or other services.

Test DNS (Should Work)

kubectl run dns-test --rm -it --image=busybox --restart=Never -n lab-secure -- \
  nslookup kubernetes.default
Expected Output 
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
pod "dns-test" deleted

DNS resolution works because the allow-dns policy permits egress to CoreDNS.

Test External Access (Should Fail)

kubectl run ext-test --rm -it --image=busybox --restart=Never -n lab-secure -- \
  wget -qO- --timeout=3 http://ifconfig.me
Expected Output 
wget: download timed out
pod "ext-test" deleted
pod lab-secure/ext-test terminated (Error)

External access is blocked. DNS resolves ifconfig.me but the TCP connection to port 80 times out because no egress rule allows it.

Test Internal Service Access (Should Fail)

First, create a target service in a different namespace:

kubectl create namespace lab-target
kubectl run web --image=nginx --port=80 -n lab-target
kubectl expose pod web --port=80 -n lab-target

Now test from lab-secure:

kubectl run svc-test --rm -it --image=busybox --restart=Never -n lab-secure -- \
  wget -qO- --timeout=3 http://web.lab-target.svc.cluster.local
Expected Output 
wget: download timed out
pod "svc-test" deleted
pod lab-secure/svc-test terminated (Error)

DNS resolves the service name, but the connection is blocked by the default deny egress policy. Only DNS egress is allowed.

Exercise 4: Namespace Isolation — Frontend → Backend → Database

Task

Create three namespaces that represent a tiered application:

  • lab-frontend → can reach lab-backend on port 8080
  • lab-backend → can reach lab-database on port 5432
  • lab-frontend → cannot reach lab-database directly

Setup

# Create namespaces
kubectl create namespace lab-frontend
kubectl create namespace lab-backend
kubectl create namespace lab-database

# Label them
kubectl label namespace lab-frontend tier=frontend
kubectl label namespace lab-backend tier=backend
kubectl label namespace lab-database tier=database

# Deploy workloads
kubectl run frontend-app --image=busybox --command -n lab-frontend -- sleep 3600
kubectl run backend-app --image=nginx --port=8080 -n lab-backend
kubectl run database-app --image=nginx --port=5432 -n lab-database

# Label the pods (in addition to the default run=<name> label)
kubectl label pod frontend-app -n lab-frontend app=frontend
kubectl label pod backend-app -n lab-backend app=backend
kubectl label pod database-app -n lab-database app=database

Step 1: Default Deny in All Three Namespaces

for ns in lab-frontend lab-backend lab-database; do
  cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: $ns
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
EOF
done

Step 2: Allow DNS in All Three Namespaces

for ns in lab-frontend lab-backend lab-database; do
  cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: $ns
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
EOF
done

Step 3: Allow Frontend → Backend

Egress from frontend:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-backend
  namespace: lab-frontend
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              tier: backend
      ports:
        - protocol: TCP
          port: 8080

Ingress to backend from frontend:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-from-frontend
  namespace: lab-backend
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              tier: frontend
      ports:
        - protocol: TCP
          port: 8080

Step 4: Allow Backend → Database

Egress from backend:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-database
  namespace: lab-backend
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              tier: database
      ports:
        - protocol: TCP
          port: 5432

Ingress to database from backend:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-from-backend
  namespace: lab-database
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              tier: backend
      ports:
        - protocol: TCP
          port: 5432

Apply all four policies:

kubectl apply -f allow-egress-to-backend.yaml
kubectl apply -f allow-ingress-from-frontend.yaml
kubectl apply -f allow-egress-to-database.yaml
kubectl apply -f allow-ingress-from-backend.yaml

Test: Frontend → Backend (Should Work)

BACKEND_IP=$(kubectl get pod backend-app -n lab-backend -o jsonpath='{.status.podIP}')

kubectl run test-fe --rm -it --image=busybox --restart=Never -n lab-frontend -- \
  wget -qO- --timeout=3 http://$BACKEND_IP:8080
Expected Output

You should see the nginx default page HTML. This confirms frontend → backend traffic on port 8080 is allowed.

Test: Frontend → Database (Should Fail)

DB_IP=$(kubectl get pod database-app -n lab-database -o jsonpath='{.status.podIP}')

kubectl run test-fe-db --rm -it --image=busybox --restart=Never -n lab-frontend -- \
  wget -qO- --timeout=3 http://$DB_IP:5432
Expected Output 
wget: download timed out

Frontend cannot reach the database directly — the egress policy in lab-frontend only allows traffic to the backend namespace, and the ingress policy in lab-database only allows traffic from the backend namespace.

Test: Backend → Database (Should Work)

kubectl run test-be --rm -it --image=busybox --restart=Never -n lab-backend -- \
  nc -zv -w 3 $DB_IP 5432
Expected Output 
<DB_IP> (DB_IP:5432) open

Backend can reach the database on port 5432.

Exercise 5: Block Cloud Metadata

Task

Create a policy in the lab-secure namespace that prevents pods from reaching the cloud metadata endpoint at 169.254.169.254.

Steps

Since we already have default deny egress in lab-secure, the metadata endpoint is already blocked. But let’s create an explicit block as an additional defense layer (useful for namespaces that might have broader egress allowed):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-cloud-metadata
  namespace: lab-secure
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32

Apply it:

kubectl apply -f block-cloud-metadata.yaml

Test

kubectl run test-metadata --rm -it --image=busybox --restart=Never -n lab-secure -- \
  wget -qO- --timeout=3 http://169.254.169.254/latest/meta-data/
Expected Output 
wget: download timed out

The metadata endpoint is unreachable. On a cloud provider (AWS/GCP/Azure), this prevents pods from accessing instance metadata and stealing IAM credentials.

Exercise 6: Test All Policies with Temporary Pods

Comprehensive Test Script

Run these commands sequentially to verify all policies are working:

echo "=== Test 1: DNS in lab-secure (expect: SUCCESS) ==="
kubectl run t1 --rm -it --image=busybox --restart=Never -n lab-secure -- \
  nslookup kubernetes.default 2>&1 | head -5

echo "=== Test 2: External access from lab-secure (expect: TIMEOUT) ==="
kubectl run t2 --rm -it --image=busybox --restart=Never -n lab-secure -- \
  sh -c "wget -qO- --timeout=3 http://1.1.1.1 2>&1 || echo BLOCKED"

echo "=== Test 3: Frontend → Backend on 8080 (expect: SUCCESS) ==="
BACKEND_IP=$(kubectl get pod backend-app -n lab-backend -o jsonpath='{.status.podIP}')
kubectl run t3 --rm -it --image=busybox --restart=Never -n lab-frontend -- \
  sh -c "wget -qO- --timeout=3 http://$BACKEND_IP:8080 2>&1 | head -3 || echo BLOCKED"

echo "=== Test 4: Frontend → Database on 5432 (expect: BLOCKED) ==="
DB_IP=$(kubectl get pod database-app -n lab-database -o jsonpath='{.status.podIP}')
kubectl run t4 --rm -it --image=busybox --restart=Never -n lab-frontend -- \
  sh -c "nc -zv -w 3 $DB_IP 5432 2>&1 || echo BLOCKED"

echo "=== Test 5: Backend → Database on 5432 (expect: SUCCESS) ==="
kubectl run t5 --rm -it --image=busybox --restart=Never -n lab-backend -- \
  sh -c "nc -zv -w 3 $DB_IP 5432 2>&1 || echo BLOCKED"

echo "=== Test 6: Metadata access from lab-secure (expect: BLOCKED) ==="
kubectl run t6 --rm -it --image=busybox --restart=Never -n lab-secure -- \
  sh -c "wget -qO- --timeout=3 http://169.254.169.254/ 2>&1 || echo BLOCKED"
Expected Results Summary

| Test | Expected Result | Why | |—|—|—| | 1. DNS in lab-secure | SUCCESS | allow-dns policy permits egress to CoreDNS | | 2. External access | BLOCKED | default deny egress, no allow for external | | 3. Frontend → Backend | SUCCESS | egress + ingress policies allow on port 8080 | | 4. Frontend → Database | BLOCKED | no egress from frontend to database namespace | | 5. Backend → Database | SUCCESS | egress + ingress policies allow on port 5432 | | 6. Metadata access | BLOCKED | default deny + block-metadata policy |

Exercise 7: Debug a Broken NetworkPolicy

Scenario

A colleague created a NetworkPolicy to allow pods labeled app: monitor in the lab-backend namespace to accept ingress from the monitoring namespace. But it is not working — traffic from monitoring is still blocked.

Here is the broken policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring
  namespace: lab-backend
spec:
  podSelector:
    matchLabels:
      app: monitor
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              team: monitoring
      ports:
        - protocol: TCP
          port: 9090

Task

Find and fix all the bugs in this policy.

Setup for Debugging

# Create monitoring namespace
kubectl create namespace monitoring
kubectl label namespace monitoring team=monitoring

# Create a target pod in lab-backend
kubectl run monitor-target --image=nginx --port=9090 -n lab-backend
kubectl label pod monitor-target -n lab-backend app=monitor

# Try to reach it from monitoring namespace
MONITOR_IP=$(kubectl get pod monitor-target -n lab-backend -o jsonpath='{.status.podIP}')
kubectl run test-mon --rm -it --image=busybox --restart=Never -n monitoring -- \
  wget -qO- --timeout=3 http://$MONITOR_IP:9090

This should timeout.

Find the Bugs

Bug 1

Missing policyTypes field.

The policy has ingress rules but no policyTypes field. While Kubernetes infers Ingress from the presence of ingress rules, it is best practice to explicitly declare it — and in some edge cases, the inference may not work as expected.

Fix:

spec:
  podSelector:
    matchLabels:
      app: monitor
  policyTypes:
    - Ingress
  ingress:
    ...
Bug 2

The monitoring namespace also needs an egress policy.

The monitoring namespace might have a default deny egress policy. If it does, traffic from monitoring to lab-backend is blocked at the source side, even though the ingress policy in lab-backend allows it.

Fix: Add an egress policy in the monitoring namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-backend
  namespace: monitoring
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              tier: backend
      ports:
        - protocol: TCP
          port: 9090
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
Bug 3

Namespace label mismatch possibility.

The policy uses namespaceSelector.matchLabels.team: monitoring. If the monitoring namespace does not have the label team=monitoring, the selector will match nothing.

Verify:

kubectl get namespace monitoring --show-labels

If the label is missing:

kubectl label namespace monitoring team=monitoring

Fixed Policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring
  namespace: lab-backend
spec:
  podSelector:
    matchLabels:
      app: monitor
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              team: monitoring
      ports:
        - protocol: TCP
          port: 9090

Apply and re-test:

kubectl apply -f allow-monitoring-fixed.yaml

kubectl run test-mon --rm -it --image=busybox --restart=Never -n monitoring -- \
  wget -qO- --timeout=3 http://$MONITOR_IP:9090

Cleanup

Remove all lab resources:

# Delete namespaces (this deletes all resources within them)
kubectl delete namespace lab-secure lab-frontend lab-backend lab-database lab-target monitoring

# Verify cleanup
kubectl get namespaces | grep lab-

Practice Questions

Question 1: In a namespace with default deny all, you apply an allow-DNS-egress policy but DNS still doesn't work. What could be wrong?

Possible causes:

  1. CoreDNS pods don’t have the label k8s-app: kube-dns — check with kubectl get pods -n kube-system --show-labels. If the label is different, update your policy’s podSelector.

  2. You only allowed UDP 53 but not TCP 53 — some DNS queries use TCP. Always allow both.

  3. The namespaceSelector doesn’t match kube-system — if you used a specific label like name: kube-system, check that the namespace actually has that label. Use kubernetes.io/metadata.name: kube-system or the empty namespaceSelector: {} to match all namespaces.

  4. The policy was applied to the wrong namespace — verify with kubectl get networkpolicy -n <namespace>.

Question 2: Write the complete set of policies to isolate a namespace called `payments` — default deny all, allow DNS, allow ingress only from `api` namespace on port 443.

Policy 1: Default deny all

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Policy 2: Allow DNS

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Policy 3: Allow ingress from api namespace

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-from-api
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: api
      ports:
        - protocol: TCP
          port: 443

Apply all three policies. The result: payments pods can only resolve DNS, and only accept ingress from the api namespace on port 443.

Question 3: How do you verify that a NetworkPolicy is actually being enforced and not silently ignored?

  1. Check the CNI plugin: kubectl get pods -n kube-system | grep -E "calico|cilium|weave". If using Flannel, policies are silently ignored.

  2. Test with a denied connection: Create a test pod in a namespace/with labels that should be denied, and attempt to connect. If it succeeds, the policy is not being enforced.

  3. Test before and after: Test connectivity before applying the policy (should succeed), then test after (should fail). If both succeed, the CNI is not enforcing policies.

# Before policy — should succeed
kubectl run pre-test --rm -it --image=busybox --restart=Never -- wget -qO- --timeout=3 http://<target>

# Apply default deny
kubectl apply -f default-deny-all.yaml

# After policy — should fail
kubectl run post-test --rm -it --image=busybox --restart=Never -- wget -qO- --timeout=3 http://<target>
Question 4: You need to allow a pod to reach an external API at 198.51.100.50 on port 443, while blocking all other egress (including cloud metadata). Write the complete policy. 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-external-api
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-consumer
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 198.51.100.50/32
      ports:
        - protocol: TCP
          port: 443
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

This allows the api-consumer pod to reach only 198.51.100.50:443 and DNS. Cloud metadata (169.254.169.254) is not in the allow list, so with default deny egress in place, it is blocked.

Question 5: In the lab, the frontend → database test was blocked. If you accidentally used OR instead of AND in the database ingress policy (separate list items for namespaceSelector and podSelector), what would happen?

OR version (separate list items):

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            tier: backend
      - podSelector:
          matchLabels:
            app: backend

This allows ingress from any pod in the backend namespace OR from any pod labeled app: backend in the same namespace (lab-database).

The OR condition is less restrictive — it allows traffic from all pods in the backend namespace regardless of their labels, and would also allow a pod labeled app: backend within the database namespace itself to connect.

AND version (same list item):

ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            tier: backend
        podSelector:
          matchLabels:
            app: backend

This only allows ingress from pods labeled app: backend that are in the backend namespace. More restrictive and more secure.

In the CKS exam, always double-check YAML indentation for AND vs OR — it is a common source of overly permissive policies.

Updated:

You may also enjoy