CKAD — Handling Job Failures

Handling Job Failures

Jobs can fail — containers crash, commands return non-zero exit codes, or tasks take too long. Kubernetes provides several mechanisms to control failure behavior.

backoffLimit

Controls how many times Kubernetes retries failed Pods before giving up:

apiVersion: batch/v1
kind: Job
metadata:
  name: retry-job
spec:
  backoffLimit: 3
  template:
    spec:
      containers:
      - name: flaky
        image: busybox:1.36
        command: ["/bin/sh", "-c", "exit 1"]
      restartPolicy: Never

After 3 failed attempts, Kubernetes stops retrying and marks the Job as failed.

kubectl apply -f retry-job.yaml

# Watch the retries
kubectl get pods -l job-name=retry-job -w

# After failures, check the Job status
kubectl describe job retry-job | grep -A 5 "Pods Statuses"

How backoffLimit Works

backoffLimit	Total Pods Created	Behavior
0	1	No retries — fail immediately
3	4 (1 original + 3 retries)	Retry 3 times with exponential backoff
6 (default)	7 (1 original + 6 retries)	Default retry behavior

Kubernetes uses exponential backoff for retries: 10s, 20s, 40s, up to 6 minutes max.

activeDeadlineSeconds

Sets a total time limit for the Job. If the Job does not finish in time, Kubernetes terminates all running Pods:

apiVersion: batch/v1
kind: Job
metadata:
  name: timed-job
spec:
  activeDeadlineSeconds: 30
  template:
    spec:
      containers:
      - name: worker
        image: busybox:1.36
        command: ["/bin/sh", "-c", "echo 'Working...' && sleep 60"]
      restartPolicy: Never

This Job will be killed after 30 seconds, even though the container tries to sleep for 60.

kubectl apply -f timed-job.yaml

# Wait 30 seconds, then check
kubectl describe job timed-job | tail -5
# Expected: "Job was active longer than specified deadline"

Important: activeDeadlineSeconds overrides backoffLimit. Even if retries remain, the Job is terminated when the deadline expires.

restartPolicy: Never vs OnFailure

Policy	Behavior	What Happens on Failure
Never	Failed Pod is left as-is. New Pod created for retry.	You get multiple Pods (good for debugging)
OnFailure	Failed container restarts in the same Pod.	Logs are lost on restart (harder to debug)

When to Use Each

# Use Never — keeps failed Pods for log inspection
spec:
  template:
    spec:
      restartPolicy: Never

# Use OnFailure — fewer Pods, but harder to debug
spec:
  template:
    spec:
      restartPolicy: OnFailure

Exam tip: Use Never when you need to inspect failed Pod logs. Use OnFailure when you want to keep Pod count low.

Auto-Cleanup with TTL

After a Job completes, the Pod sticks around (in Completed status) so you can check logs. Use ttlSecondsAfterFinished to auto-delete:

apiVersion: batch/v1
kind: Job
metadata:
  name: auto-cleanup-job
spec:
  ttlSecondsAfterFinished: 60
  template:
    spec:
      containers:
      - name: task
        image: busybox:1.36
        command: ["/bin/sh", "-c", "echo done"]
      restartPolicy: Never

60 seconds after the Job finishes, both the Job and its Pods are deleted automatically.

TTL Behavior

ttlSecondsAfterFinished	Behavior
Not set (default)	Job and Pods remain until manually deleted
0	Delete immediately after completion
60	Delete 60 seconds after completion
3600	Delete 1 hour after completion

Failure Handling Summary

Mechanism	Controls	Example
backoffLimit	Number of retry attempts	backoffLimit: 3 — give up after 3 retries
activeDeadlineSeconds	Total time allowed	activeDeadlineSeconds: 300 — kill after 5 min
restartPolicy	How failed containers restart	Never = new Pod, OnFailure = same Pod
ttlSecondsAfterFinished	When to clean up after done	60 — delete 1 min after Job finishes

---

Next: Learn about scheduled tasks with CronJobs →