CKS — Image Security Lab

12 minute read

Image Security — Hands-On Lab

CKS Exam Domain: Supply Chain Security (20%)

This lab walks you through the complete image security lifecycle — scanning for vulnerabilities, building minimal images, enforcing registry restrictions, signing images, and configuring ImagePolicyWebhook. Each exercise mirrors CKS exam patterns.

Prerequisites

  • A Kubernetes cluster with control plane access (kubeadm preferred)
  • kubectl configured with cluster-admin access
  • trivy installed (brew install trivy or apt install trivy)
  • cosign installed (brew install cosign)
  • docker or podman available for building images
  • Access to a container registry (Docker Hub, GHCR, or a private registry)

Lab 1: Scan Images with Trivy

Task

Scan container images at different severity levels, identify vulnerable images in a running cluster, and take remediation action.

Steps

# Create a namespace for the lab
kubectl create namespace image-lab

# Deploy Pods with various images
kubectl run nginx-old --image=nginx:1.21 -n image-lab
kubectl run nginx-new --image=nginx:1.27 -n image-lab
kubectl run redis --image=redis:7.2 -n image-lab
kubectl run alpine --image=alpine:3.19 -n image-lab

# Wait for Pods to be running
kubectl get pods -n image-lab -w

Scan All Images

# Step 1: Get all unique images in the namespace
kubectl get pods -n image-lab -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u

# Expected output:
# alpine:3.19
# nginx:1.21
# nginx:1.27
# redis:7.2

Scan with Different Severity Filters

# Full scan (all severities)
trivy image nginx:1.21

# CRITICAL only
trivy image --severity CRITICAL nginx:1.21

# CRITICAL and HIGH
trivy image --severity CRITICAL,HIGH nginx:1.21

# Quiet mode (summary only)
trivy image --severity CRITICAL --quiet nginx:1.21

Find and Delete Vulnerable Pods

# Scan each image and identify CRITICAL vulnerabilities
for img in $(kubectl get pods -n image-lab -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u); do
  echo "=== Scanning: $img ==="
  if ! trivy image --severity CRITICAL --exit-code 1 --quiet "$img" 2>/dev/null; then
    echo ">>> CRITICAL vulnerabilities found in: $img"
    # Find Pods using this image
    echo "Affected Pods:"
    kubectl get pods -n image-lab -o json | \
      jq -r --arg IMG "$img" '.items[] | select(.spec.containers[].image == $IMG) | .metadata.name'
  fi
  echo ""
done

Delete Pods with CRITICAL Vulnerabilities

# Example: If nginx:1.21 has CRITICAL CVEs, delete the Pod
kubectl delete pod nginx-old -n image-lab

# Verify
kubectl get pods -n image-lab

Expected Output

The older nginx:1.21 image will have significantly more CRITICAL vulnerabilities than nginx:1.27. The alpine:3.19 image will typically have zero or very few.

Lab 2: Build a Minimal Multi-Stage Image

Task

Take a Python application with a bloated Dockerfile and rebuild it using a multi-stage build with a distroless runtime image.

Step 1: Create the Application

mkdir -p /tmp/image-lab/app && cd /tmp/image-lab

# Create a simple Python app
cat > app/main.py <<'EOF'
from http.server import HTTPServer, BaseHTTPRequestHandler
import json

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        response = {"status": "ok", "message": "Secure minimal image"}
        self.wfile.write(json.dumps(response).encode())

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8080), Handler)
    print("Server running on port 8080")
    server.serve_forever()
EOF

Step 2: Create the Bloated Dockerfile (Before)

cat > Dockerfile.bloated <<'EOF'
FROM python:3.12
WORKDIR /app
COPY . /app
RUN pip install flask requests boto3
ENV SECRET_KEY=mysecretkey123
CMD ["python", "app/main.py"]
EOF

Step 3: Create the Minimal Dockerfile (After)

cat > Dockerfile.minimal <<'EOF'
# ──────────────────────────
# Stage 1: Build
# ──────────────────────────
FROM python:3.12-slim AS builder
WORKDIR /build
# No external dependencies needed for this app
# If needed: COPY requirements.txt . && pip install --prefix=/install -r requirements.txt

# ──────────────────────────
# Stage 2: Runtime
# ──────────────────────────
FROM gcr.io/distroless/python3-debian12

COPY app/main.py /app/main.py

WORKDIR /app
USER 65534

CMD ["main.py"]
EOF

Step 4: Build and Compare

# Build both images
docker build -f Dockerfile.bloated -t myapp:bloated .
docker build -f Dockerfile.minimal -t myapp:minimal .

# Compare sizes
docker images | grep myapp

Expected Output

REPOSITORY   TAG       IMAGE ID       SIZE
myapp        bloated   a1b2c3d4e5f6   1.02GB
myapp        minimal   f6e5d4c3b2a1   52.8MB

Step 5: Scan Both Images

echo "=== Bloated Image ==="
trivy image --severity CRITICAL,HIGH myapp:bloated

echo ""
echo "=== Minimal Image ==="
trivy image --severity CRITICAL,HIGH myapp:minimal

The bloated image will have significantly more vulnerabilities due to the full Debian OS, pip packages, and development tools.

Step 6: Test Shell Access

# Bloated image — shell available (security risk)
docker run --rm -it myapp:bloated /bin/bash
# root@container:/app# whoami → root
# root@container:/app# apt list --installed | wc -l → hundreds of packages

# Minimal image — no shell (secure)
docker run --rm -it myapp:minimal /bin/sh
# docker: Error response from daemon: OCI runtime create failed:
# ... executable file not found in $PATH

Lab 3: OPA Gatekeeper — Restrict Image Registries

Task

Install OPA Gatekeeper and create a policy that allows only images from docker.io/library/ (official Docker images) and a hypothetical private registry.

Step 1: Install Gatekeeper

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.15.0/deploy/gatekeeper.yaml

# Wait for Gatekeeper to be ready
kubectl wait --for=condition=Ready pods -l control-plane=controller-manager -n gatekeeper-system --timeout=120s

Step 2: Create ConstraintTemplate

cat <<'EOF' | kubectl apply -f -
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              type: array
              items:
                type: string
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8sallowedrepos

      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        not startswith(container.image, input.parameters.repos[_])
        msg := sprintf("Container '%v' uses image '%v' not from allowed registries: %v",
          [container.name, container.image, input.parameters.repos])
      }

      violation[{"msg": msg}] {
        container := input.review.object.spec.initContainers[_]
        not startswith(container.image, input.parameters.repos[_])
        msg := sprintf("Init container '%v' uses image '%v' not from allowed registries: %v",
          [container.name, container.image, input.parameters.repos])
      }
EOF

Step 3: Create Constraint

cat <<'EOF' | kubectl apply -f -
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: allowed-repos-lab
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaces:
    - image-lab
  parameters:
    repos:
    - "docker.io/library/"
    - "registry.company.com/"
    - "nginx"
    - "redis"
    - "alpine"
EOF

Note: Images without a registry prefix (like nginx:1.27) resolve to docker.io/library/nginx:1.27. Gatekeeper matches on the literal image string in the Pod spec, so we also include the short form.

Step 4: Test the Policy

# Should SUCCEED — official image
kubectl run test-allowed --image=nginx:1.27 -n image-lab
kubectl delete pod test-allowed -n image-lab

# Should be DENIED — untrusted registry
kubectl run test-denied --image=registry.evil.com/malware:latest -n image-lab

Expected Output for Denied Request

Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request:
[allowed-repos-lab] Container 'test-denied' uses image 'registry.evil.com/malware:latest'
not from allowed registries: ["docker.io/library/", "registry.company.com/", "nginx", "redis", "alpine"]

Step 5: Check Gatekeeper Audit

# View constraint violations (audit mode finds existing violations)
kubectl get k8sallowedrepos allowed-repos-lab -o yaml | grep -A20 status

Lab 4: Test Pod Rejection from Untrusted Registry

Task

Deploy Pods from various registries and verify that only trusted sources are allowed.

Steps

# Test matrix — verify each one is allowed or denied

# ✅ Should be allowed (official nginx)
kubectl run t1 --image=nginx:1.27 -n image-lab && kubectl delete pod t1 -n image-lab
echo "Test 1: PASSED (allowed)"

# ❌ Should be denied (untrusted registry)
kubectl run t2 --image=gcr.io/random-project/app:v1 -n image-lab 2>&1 | grep -q "denied" && \
  echo "Test 2: PASSED (denied)" || echo "Test 2: FAILED"

# ❌ Should be denied (Docker Hub non-official image)
kubectl run t3 --image=someuser/someimage:latest -n image-lab 2>&1 | grep -q "denied" && \
  echo "Test 3: PASSED (denied)" || echo "Test 3: FAILED"

# ✅ Should be allowed (redis - in our allowlist)
kubectl run t4 --image=redis:7.2 -n image-lab && kubectl delete pod t4 -n image-lab
echo "Test 4: PASSED (allowed)"

Lab 5: Sign and Verify an Image with Cosign

Task

Generate a key pair, sign a container image, verify the signature, and demonstrate failed verification on an unsigned image.

Step 1: Generate Keys

cd /tmp/image-lab

# Generate key pair
cosign generate-key-pair
# Enter a password when prompted
# Creates: cosign.key, cosign.pub

ls -la cosign.*

Step 2: Tag and Push an Image

# Tag the minimal image for a registry you can push to
# (Replace with your actual registry)
docker tag myapp:minimal docker.io/<your-username>/myapp:v1.0-signed
docker push docker.io/<your-username>/myapp:v1.0-signed

Step 3: Sign the Image

cosign sign --key cosign.key docker.io/<your-username>/myapp:v1.0-signed
# Enter the password for cosign.key

# Cosign stores the signature as a separate tag in the registry

Step 4: Verify the Signature

# Verify with the public key — should SUCCEED
cosign verify --key cosign.pub docker.io/<your-username>/myapp:v1.0-signed

# Output:
# Verification for docker.io/<your-username>/myapp:v1.0-signed --
# The following checks were performed on each of these signatures:
#   - The cosign claims were validated
#   - The signatures were verified against the specified public key

Step 5: Verify an Unsigned Image — Should FAIL

# Try to verify an image that was never signed
cosign verify --key cosign.pub nginx:1.27

# Output:
# Error: no matching signatures:
# ...

Step 6: View Signature Details

# Inspect the signature payload
cosign verify --key cosign.pub docker.io/<your-username>/myapp:v1.0-signed | jq .

# Show the signature in the registry
cosign triangulate docker.io/<your-username>/myapp:v1.0-signed
# Shows the tag where the signature is stored: sha256-<digest>.sig

Lab 6: Configure ImagePolicyWebhook (Conceptual Walkthrough)

Task

Walk through the complete configuration of ImagePolicyWebhook on a kubeadm cluster. This is a conceptual exercise — on a real cluster, you would execute each command on the control plane node.

Step 1: Understand the File Structure

/etc/kubernetes/admission/
├── admission-config.yaml          ← AdmissionConfiguration
├── webhook-kubeconfig.yaml        ← Backend connection details
└── certs/
    ├── ca.crt                     ← CA certificate
    ├── ca.key                     ← CA private key
    ├── server.crt                 ← Backend server certificate
    ├── server.key                 ← Backend server key
    ├── apiserver-client.crt       ← API server client certificate
    └── apiserver-client.key       ← API server client key

Step 2: Create AdmissionConfiguration

# /etc/kubernetes/admission/admission-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
  configuration:
    imagePolicy:
      kubeConfigFile: /etc/kubernetes/admission/webhook-kubeconfig.yaml
      allowTTL: 50
      denyTTL: 50
      retryBackoff: 500
      defaultAllow: false       # Fail-closed — secure default

Step 3: Create Webhook Kubeconfig

# /etc/kubernetes/admission/webhook-kubeconfig.yaml
apiVersion: v1
kind: Config
clusters:
- name: image-policy-backend
  cluster:
    server: https://image-policy-backend.default.svc:8443/image-policy
    certificate-authority: /etc/kubernetes/admission/certs/ca.crt
users:
- name: kube-apiserver
  user:
    client-certificate: /etc/kubernetes/admission/certs/apiserver-client.crt
    client-key: /etc/kubernetes/admission/certs/apiserver-client.key
contexts:
- name: default
  context:
    cluster: image-policy-backend
    user: kube-apiserver
current-context: default

Step 4: Modify API Server Manifest

The following flags and volume mounts need to be added to /etc/kubernetes/manifests/kube-apiserver.yaml:

# Command additions:
- --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
- --admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml

# Volume mount addition:
volumeMounts:
- name: admission-config
  mountPath: /etc/kubernetes/admission
  readOnly: true

# Volume addition:
volumes:
- name: admission-config
  hostPath:
    path: /etc/kubernetes/admission
    type: DirectoryOrCreate

Step 5: Verify Configuration

# After the API server restarts, test:

# With defaultAllow: false and no backend, all Pods are denied
kubectl run test --image=nginx:1.27
# Expected: Error from server (Forbidden): ...

# Check API server logs for configuration status
sudo crictl logs $(sudo crictl ps --name kube-apiserver -q) 2>&1 | grep -i "imagepolicy"

Key Takeaways

  1. Three files are needed: AdmissionConfiguration, webhook kubeconfig, TLS certs
  2. Two API server flags: --enable-admission-plugins and --admission-control-config-file
  3. Volume mounts are required to make config available inside the static Pod
  4. defaultAllow: false is the secure default (fail-closed)
  5. The API server restarts automatically after manifest changes — wait for it

Cleanup

# Delete the lab namespace
kubectl delete namespace image-lab

# Remove Gatekeeper (if you want to clean up)
kubectl delete -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.15.0/deploy/gatekeeper.yaml

# Remove Gatekeeper CRDs
kubectl delete constrainttemplate k8sallowedrepos

# Clean up local files
rm -rf /tmp/image-lab

Practice Questions

Question 1

You are given a namespace critical-apps and asked to find all images with CRITICAL vulnerabilities and delete the affected Pods. A new image registry.company.com/api:v4.2 is provided as the replacement. Write the complete sequence of commands.

Show Answer 
# 1. List all images in the namespace
kubectl get pods -n critical-apps -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{"\n"}{end}{end}'

# 2. Scan each unique image
for img in $(kubectl get pods -n critical-apps -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u); do
  echo "=== $img ==="
  trivy image --severity CRITICAL --quiet "$img"
done

# 3. Delete Pods running images with CRITICAL CVEs
for img in $(kubectl get pods -n critical-apps -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u); do
  if ! trivy image --severity CRITICAL --exit-code 1 --quiet "$img" 2>/dev/null; then
    echo "Deleting Pods with image: $img"
    kubectl get pods -n critical-apps -o json | \
      jq -r --arg IMG "$img" '.items[] | select(.spec.containers[].image == $IMG) | .metadata.name' | \
      xargs kubectl delete pod -n critical-apps
  fi
done

# 4. If a Deployment controls these Pods, update the image
kubectl set image deployment/<deployment-name> <container-name>=registry.company.com/api:v4.2 -n critical-apps

# 5. Verify the rollout
kubectl rollout status deployment/<deployment-name> -n critical-apps

# 6. Scan the replacement image
trivy image --severity CRITICAL registry.company.com/api:v4.2

Question 2

Convert the following Dockerfile into a secure multi-stage build:

FROM ubuntu:22.04
RUN apt update && apt install -y golang-go
COPY . /app
WORKDIR /app
RUN go build -o server .
CMD ["./server"]
Show Answer 
# Stage 1: Build
FROM golang:1.22-alpine AS builder
WORKDIR /build
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o server .

# Stage 2: Runtime
FROM gcr.io/distroless/static-debian12
COPY --from=builder /build/server /server
USER 65534
ENTRYPOINT ["/server"]

Improvements:

  1. Multi-stage build — Go compiler and source not in final image
  2. CGO_ENABLED=0 — Static binary, no C dependencies
  3. -ldflags="-s -w" — Strips debug info, smaller binary
  4. distroless/static — No shell, no package manager (~2 MB base)
  5. USER 65534 — Runs as non-root (nobody)
  6. Final image: ~10 MB vs ~800 MB+ with ubuntu+golang

Question 3

Write a Gatekeeper Constraint that denies Pods in the production namespace if they use the latest tag.

Show Answer

First, the ConstraintTemplate:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdisallowedtags
spec:
  crd:
    spec:
      names:
        kind: K8sDisallowedTags
      validation:
        openAPIV3Schema:
          type: object
          properties:
            tags:
              type: array
              items:
                type: string
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8sdisallowedtags

      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        tag := split(container.image, ":")[1]
        tag == input.parameters.tags[_]
        msg := sprintf("Container '%v' uses disallowed tag '%v' in image '%v'",
          [container.name, tag, container.image])
      }

      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        not contains(container.image, ":")
        not contains(container.image, "@")
        msg := sprintf("Container '%v' image '%v' has no tag (defaults to 'latest')",
          [container.name, container.image])
      }

Then the Constraint:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDisallowedTags
metadata:
  name: no-latest-tag
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaces:
    - production
  parameters:
    tags:
    - "latest"

Question 4

During the CKS exam, you are asked to configure ImagePolicyWebhook with defaultAllow: false. The backend webhook URL is https://image-checker.default.svc:1323/image_policy. The CA, client cert, and client key are at /etc/kubernetes/admission/certs/. Write the AdmissionConfiguration and webhook kubeconfig.

Show Answer

AdmissionConfiguration (/etc/kubernetes/admission/admission-config.yaml):

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
  configuration:
    imagePolicy:
      kubeConfigFile: /etc/kubernetes/admission/webhook-kubeconfig.yaml
      allowTTL: 50
      denyTTL: 50
      retryBackoff: 500
      defaultAllow: false

Webhook kubeconfig (/etc/kubernetes/admission/webhook-kubeconfig.yaml):

apiVersion: v1
kind: Config
clusters:
- name: image-checker
  cluster:
    server: https://image-checker.default.svc:1323/image_policy
    certificate-authority: /etc/kubernetes/admission/certs/ca.crt
users:
- name: api-server
  user:
    client-certificate: /etc/kubernetes/admission/certs/apiserver-client.crt
    client-key: /etc/kubernetes/admission/certs/apiserver-client.key
contexts:
- name: default
  context:
    cluster: image-checker
    user: api-server
current-context: default

API server flags:

--enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
--admission-control-config-file=/etc/kubernetes/admission/admission-config.yaml

Question 5

What is the complete defense-in-depth strategy for image security in Kubernetes? List each layer in order.

Show Answer
Order Layer Action Tool
1 Build Use minimal base images (distroless, Alpine), multi-stage builds, non-root user, no secrets in layers Dockerfile best practices
2 Scan Scan for CVEs before pushing to registry, fail CI on CRITICAL Trivy, Grype
3 Sign Sign images after build, create provenance attestations Cosign, Sigstore
4 Store Use private registries, enable vulnerability scanning at registry level Harbor, ECR, GCR
5 Admit Enforce registry restrictions, require signatures, validate policies OPA Gatekeeper, Kyverno, ImagePolicyWebhook
6 Deploy Pin images by digest, use imagePullPolicy: Always, set imagePullSecrets Kubernetes Pod spec
7 Runtime Monitor for anomalous behavior, detect drift from original image Falco, runtime scanners
8 Continuously Scan Re-scan running images as new CVEs are disclosed Trivy operator, scheduled scans

No single control is sufficient. Each layer catches what the previous one missed.

Updated:

You may also enjoy