janak0ff

Posted on Aug 13

Create Countdown Job in Kubernetes

The Nautilus DevOps team is crafting jobs in the Kubernetes cluster. While they're developing actual scripts/commands, they're currently setting up templates and testing jobs with dummy commands. Please create a job template as per details given below:

Create a job named countdown-xfusion.
The spec template should be named countdown-xfusion (under metadata), and the container should be named container-countdown-xfusion
Utilize image debian with latest tag (ensure to specify as debian:latest), and set the restart policy to Never.
Execute the command sleep 5

Kubernetes Jobs: A Beginner's Guide to Batch Processing

Introduction

Imagine you need to run a one-time database migration, process a large dataset, or perform a system cleanup. You don't want this task running forever like a web server—you just want it to run, complete, and stop. This is exactly what Kubernetes Jobs are designed for!

In this beginner-friendly guide, we'll walk through creating and managing Jobs in Kubernetes. By the end, you'll be able to run any batch processing task in your cluster with confidence.

What You'll Learn

What Kubernetes Jobs are and when to use them
How to create and configure Jobs
How Jobs differ from Pods and Deployments
How to monitor and troubleshoot Jobs
Best practices for production use

What are Kubernetes Jobs?
Understanding the Problem Jobs Solve
Step-by-Step Job Creation
Job Lifecycle and States
Monitoring and Verification
Advanced Job Configuration
Real-World Examples
Common Issues and Troubleshooting
Best Practices
Conclusion

What are Kubernetes Jobs?

The Definition

A Job is a Kubernetes resource that creates one or more pods and ensures that a specified number of them successfully terminate. Think of it as a task runner that:

Starts a pod
Runs a command or script
Waits for it to complete
Reports success or failure

The Problem Jobs Solve

In any infrastructure, you need to run tasks that:

Run once and stop - Not continuous like web servers
Process data - Batch processing, ETL jobs
Perform migrations - Database schema updates
Run maintenance - Cleanup, backups, system checks

Without Jobs, you'd have to:

Run scripts manually (error-prone)
Use cron (limited scalability)
Keep pods running indefinitely (waste resources)

How Jobs Work

┌─────────────────────────────────────────────┐
│              Job                            │
│         (countdown-xfusion)                 │
│         COMPLETIONS: 1/1                    │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│              Pod                            │
│         (countdown-xfusion-xxxxx)           │
│         STATUS: Completed                   │
│                                             │
│  ┌─────────────────────────────────────┐   │
│  │   Container                         │   │
│  │   Image: debian:latest              │   │
│  │   Command: sleep 5                  │   │
│  │   STATUS: Exited (0)                │   │
│  └─────────────────────────────────────┘   │
└─────────────────────────────────────────────┘

Understanding the Problem Jobs Solve

Job vs Pod vs Deployment

Resource	Purpose	Runs Until	Restart Policy	Use Case
Pod	Run a container	Pod terminates	Always	Simple tasks
Job	Run to completion	Success/failure	Never/OnFailure	Batch processing
CronJob	Run on schedule	Each run completes	Never/OnFailure	Scheduled tasks
Deployment	Continuous service	Never	Always	Web servers, APIs
StatefulSet	Stateful service	Never	Always	Databases

When to Use a Job

✅ Use a Job when:

You need to run a task once
The task should complete (not run forever)
You want to ensure the task succeeds
You need to process a batch of data
You're doing database migrations
You're running system maintenance

❌ Don't use a Job when:

Your application needs to run continuously
You need auto-scaling
You need rolling updates
You need to expose a service

Step-by-Step Job Creation

The Scenario

The Nautilus DevOps team needs to create a Job that:

Runs once and completes
Uses the debian:latest image
Executes the command: sleep 5
Has restart policy: Never

Step 1: Generate the YAML

First, let's create the base YAML manifest:

kubectl create job countdown-xfusion \
  --image=debian:latest \
  --dry-run=client -o yaml > countdown-job.yaml

This generates the YAML template for us.

Step 2: Edit the YAML

Open the file and customize it:

nano countdown-job.yaml

Before (generated):

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  name: countdown-xfusion
spec:
  template:
    metadata:
      creationTimestamp: null
    spec:
      containers:
      - image: debian:latest
        name: countdown-xfusion
        resources: {}
      restartPolicy: Never
status: {}

After (modified):

apiVersion: batch/v1
kind: Job
metadata:
  name: countdown-xfusion
spec:
  template:
    metadata:
      name: countdown-xfusion
    spec:
      containers:
      - name: container-countdown-xfusion
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "sleep 5"]
      restartPolicy: Never

Step 3: Apply the Job

Create the Job in your cluster:

kubectl apply -f countdown-job.yaml

Expected output:

job.batch/countdown-xfusion created

Step 4: Verify the Job

Check that the Job was created:

kubectl get jobs

Expected output:

NAME                COMPLETIONS   DURATION   AGE
countdown-xfusion   0/1           2s         5s

Step 5: Check Pods

View the pods created by the Job:

kubectl get pods

Expected output:

NAME                      READY   STATUS      RESTARTS   AGE
countdown-xfusion-xxxxx   0/1     Completed   0          10s

Step 6: View Logs

Check the output (sleep 5 produces no output):

kubectl logs countdown-xfusion-xxxxx
# No output (expected)

Step 7: Check Completion

Wait a few seconds and check again:

kubectl get jobs

Expected output:

NAME                COMPLETIONS   DURATION   AGE
countdown-xfusion   1/1           5s         15s

Job Lifecycle and States

Job Lifecycle Diagram

┌─────────────────────────────────────────────────────────────┐
│                     Job Lifecycle                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────┐     ┌──────────┐     ┌──────────┐           │
│  │ Created  │────▶│ Pending  │────▶│ Running  │           │
│  └──────────┘     └──────────┘     └────┬─────┘           │
│                                         │                  │
│                         ┌───────────────┼───────────────┐  │
│                         │               │               │  │
│                         ▼               ▼               ▼  │
│                    ┌──────────┐   ┌──────────┐   ┌──────────┐
│                    │Complete  │   │  Failed  │   │  Retry   │
│                    │(Success) │   │  (Error) │   │          │
│                    └──────────┘   └──────────┘   └──────────┘
│                                                             │
└─────────────────────────────────────────────────────────────┘

Job States

State	Description	Indicates
Pending	Waiting for resources	Scheduling in progress
Running	Pods are executing	Task is in progress
Complete	All pods succeeded	Task finished successfully ✅
Failed	Some pods failed	Task encountered errors ❌

Pod States

State	Description
Pending	Waiting to be scheduled
Running	Container is executing
Completed	Container exited with 0 (success)
Error	Container exited with non-zero
CrashLoopBackOff	Container keeps crashing

Monitoring and Verification

1. Check Job Status

# Basic status
kubectl get job countdown-xfusion

# Detailed status
kubectl describe job countdown-xfusion

# Check completion status
kubectl get job countdown-xfusion -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}'

# Check if failed
kubectl get job countdown-xfusion -o jsonpath='{.status.conditions[?(@.type=="Failed")].status}'

2. View Pods

# List all pods
kubectl get pods

# Filter by job name
kubectl get pods -l job-name=countdown-xfusion

# Get pod details
kubectl describe pod -l job-name=countdown-xfusion

# Pod status
kubectl get pod -l job-name=countdown-xfusion -o jsonpath='{.items[0].status.phase}'

3. Check Logs

# Get pod name
POD_NAME=$(kubectl get pods -l job-name=countdown-xfusion -o name | cut -d'/' -f2)

# View logs
kubectl logs $POD_NAME

# If command produced output
kubectl logs $POD_NAME

# View previous logs (if pod restarted)
kubectl logs $POD_NAME --previous

4. Check Events

# Events related to job
kubectl get events --field-selector involvedObject.name=countdown-xfusion

# All events
kubectl get events --sort-by='.lastTimestamp' | tail -10

# Events related to pods
kubectl get events --field-selector involvedObject.kind=Pod | grep countdown-xfusion

5. Complete Verification Script

#!/bin/bash
echo "=== Job Verification ==="

echo -e "\n1. Job Status:"
kubectl get job countdown-xfusion

echo -e "\n2. Pods:"
kubectl get pods -l job-name=countdown-xfusion

echo -e "\n3. Job Details:"
kubectl describe job countdown-xfusion | grep -E "Name:|Namespace:|Completions:|Duration:|Status:"

echo -e "\n4. Container Details:"
CONTAINER_NAME=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.containers[0].name}')
IMAGE=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.containers[0].image}')
COMMAND=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.containers[0].command}')
ARGS=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.containers[0].args}')
RESTART=$(kubectl get job countdown-xfusion -o jsonpath='{.spec.template.spec.restartPolicy}')

echo "Container Name: $CONTAINER_NAME"
echo "Image: $IMAGE"
echo "Command: $COMMAND"
echo "Args: $ARGS"
echo "Restart Policy: $RESTART"

echo -e "\n5. Pod Logs:"
POD_NAME=$(kubectl get pods -l job-name=countdown-xfusion -o name | cut -d'/' -f2)
if [ -n "$POD_NAME" ]; then
    echo "Logs from $POD_NAME:"
    kubectl logs $POD_NAME 2>/dev/null || echo "No logs (command produced no output)"
else
    echo "No pods found"
fi

Advanced Job Configuration

1. Parallel Jobs

Run multiple pods in parallel:

apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-job
spec:
  completions: 5        # Total pods to complete
  parallelism: 2        # Run 2 pods at a time
  template:
    spec:
      containers:
      - name: worker
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "echo Working; sleep 10"]
      restartPolicy: Never

2. Job with Failure Handling

apiVersion: batch/v1
kind: Job
metadata:
  name: fault-tolerant-job
spec:
  backoffLimit: 5              # Retry up to 5 times
  activeDeadlineSeconds: 60    # Max 60 seconds
  template:
    spec:
      containers:
      - name: worker
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "sleep 5; exit 1"]  # Intentional failure
      restartPolicy: Never

3. Job with Resource Limits

apiVersion: batch/v1
kind: Job
metadata:
  name: resource-job
spec:
  template:
    spec:
      containers:
      - name: worker
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "echo Processing; sleep 5"]
        resources:
          requests:
            memory: "32Mi"
            cpu: "50m"
          limits:
            memory: "64Mi"
            cpu: "100m"
      restartPolicy: Never

4. Auto-Cleanup After Completion

apiVersion: batch/v1
kind: Job
metadata:
  name: auto-cleanup-job
spec:
  ttlSecondsAfterFinished: 60   # Delete after 60 seconds
  template:
    spec:
      containers:
      - name: worker
        image: debian:latest
        command: ["/bin/bash"]
        args: ["-c", "echo 'Job done!'; sleep 5"]
      restartPolicy: Never

5. Complete Production-Ready Job

apiVersion: batch/v1
kind: Job
metadata:
  name: countdown-xfusion
  namespace: production
  labels:
    app: countdown
    type: job
    environment: production
    team: nautilus
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  activeDeadlineSeconds: 30
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      name: countdown-xfusion
      labels:
        app: countdown
        type: job
    spec:
      containers:
      - name: container-countdown-xfusion
        image: debian:bullseye-slim
        imagePullPolicy: IfNotPresent
        command: ["/bin/bash"]
        args: ["-c", "sleep 5"]
        resources:
          requests:
            memory: "16Mi"
            cpu: "50m"
          limits:
            memory: "32Mi"
            cpu: "100m"
      restartPolicy: Never

Real-World Examples

Example 1: Database Migration

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  backoffLimit: 1
  template:
    spec:
      containers:
      - name: migration
        image: postgres:13
        command: ["/bin/sh"]
        args: ["-c", "psql -h db-service -U postgres -f /migrations/schema.sql"]
        env:
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: password
        volumeMounts:
        - name: migrations
          mountPath: /migrations
      restartPolicy: Never
      volumes:
      - name: migrations
        configMap:
          name: db-migrations

Example 2: Data Processing

apiVersion: batch/v1
kind: Job
metadata:
  name: data-processor
spec:
  completions: 10
  parallelism: 3
  template:
    spec:
      containers:
      - name: processor
        image: python:3.9
        command: ["/bin/sh"]
        args: ["-c", "python /scripts/process.py --batch=$BATCH_INDEX"]
        env:
        - name: BATCH_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch-index']
        volumeMounts:
        - name: scripts
          mountPath: /scripts
        - name: data
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: scripts
        configMap:
          name: data-processor-scripts
      - name: data
        persistentVolumeClaim:
          claimName: data-pvc

Example 3: System Cleanup

apiVersion: batch/v1
kind: Job
metadata:
  name: system-cleanup
spec:
  backoffLimit: 2
  template:
    spec:
      containers:
      - name: cleanup
        image: busybox:1.35
        command: ["/bin/sh"]
        args: ["-c", "find /tmp -name '*.tmp' -mtime +7 -delete"]
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      restartPolicy: Never
      volumes:
      - name: tmp
        hostPath:
          path: /tmp

Common Issues and Troubleshooting

Issue 1: Pod Not Starting

Symptoms: Job created but no pods running

Solutions:

# Check job status
kubectl describe job countdown-xfusion

# Check pod events
kubectl describe pod -l job-name=countdown-xfusion

# Check if there are resource constraints
kubectl get nodes

# Check if image exists
kubectl run test --image=debian:latest --rm -it -- /bin/bash

Issue 2: Command Not Found

Error: /bin/bash: command not found

Solutions:

# Check if bash exists in the image
kubectl run test --image=debian:latest --rm -it -- which bash

# Use sh instead
kubectl patch job countdown-xfusion -p '{"spec":{"template":{"spec":{"containers":[{"name":"container-countdown-xfusion","command":["/bin/sh","-c","sleep 5"]}]}}}}'

Issue 3: Container Exits with Error

Symptoms: Pod shows Error status

Solutions:

# Check logs
kubectl logs countdown-xfusion-xxxxx

# Check exit code
kubectl get pod countdown-xfusion-xxxxx -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'

# Check if command is correct
kubectl get job countdown-xfusion -o yaml | grep -A 5 command

# Test the command manually
kubectl run test --image=debian:latest --rm -it -- /bin/bash -c "sleep 5"

Issue 4: Job Never Completes

Symptoms: Job stuck in Running state

Solutions:

# Check if command hangs
kubectl logs countdown-xfusion-xxxxx

# Set a deadline
kubectl patch job countdown-xfusion -p '{"spec":{"activeDeadlineSeconds":30}}'

# Check pod status
kubectl describe pod -l job-name=countdown-xfusion

# Delete the job
kubectl delete job countdown-xfusion

Issue 5: Too Many Failed Attempts

Symptoms: Many failed pods

Solutions:

# Check backoff limit
kubectl get job countdown-xfusion -o jsonpath='{.spec.backoffLimit}'

# Increase backoff limit if needed
kubectl patch job countdown-xfusion -p '{"spec":{"backoffLimit":5}}'

# Clean up failed jobs
kubectl delete pods -l job-name=countdown-xfusion

Best Practices

✅ DO's

1. Use Specific Image Tags

# BAD - Unpredictable
image: debian:latest

# GOOD - Specific version
image: debian:bullseye-slim

2. Set Resource Limits

resources:
  requests:
    memory: "16Mi"
    cpu: "50m"
  limits:
    memory: "32Mi"
    cpu: "100m"

3. Set Appropriate Backoff Limit

backoffLimit: 3  # Reasonable retry limit

4. Use RestartPolicy: Never

restartPolicy: Never  # For jobs that should run once

5. Test Commands Locally

# Test before deploying
docker run --rm debian:latest /bin/bash -c "sleep 5"

6. Add Labels

metadata:
  labels:
    app: countdown
    type: job
    team: nautilus

7. Set TTL for Auto-Cleanup

ttlSecondsAfterFinished: 3600  # Clean up after 1 hour

8. Monitor Job Completion

# Set up alerts for failures
# Check job status regularly

❌ DON'Ts

1. Don't Use :latest in Production

Can cause unexpected behavior
Harder to debug

2. Don't Use RestartPolicy: Always

Jobs should use Never or OnFailure
Always will cause infinite retries

3. Don't Ignore Failed Jobs

Investigate failures
Clean up failed jobs

4. Don't Create Jobs Without Resource Limits

Can consume too many resources
May affect other workloads

5. Don't Forget to Clean Up

Completed jobs can accumulate
Use TTL or manual cleanup

Job Management Commands

Create Job

# Create from YAML
kubectl apply -f job.yaml

# Create with command
kubectl create job NAME --image=IMAGE

View Jobs

# List jobs
kubectl get jobs

# List jobs in all namespaces
kubectl get jobs --all-namespaces

# Get job details
kubectl describe job NAME

# View job YAML
kubectl get job NAME -o yaml

Update Job

# Edit job
kubectl edit job NAME

# Patch job
kubectl patch job NAME -p '{"spec":{"backoffLimit":5}}'

Delete Job

# Delete job
kubectl delete job NAME

# Delete using YAML
kubectl delete -f job.yaml

# Delete with force
kubectl delete job NAME --force --grace-period=0

View Pods

# View pods for job
kubectl get pods -l job-name=NAME

# View pod logs
kubectl logs POD_NAME

# View pod details
kubectl describe pod POD_NAME

Quick Reference Card

Job Commands

Command	Description
`kubectl create job NAME --image=IMAGE`	Create a job
`kubectl get jobs`	List jobs
`kubectl describe job NAME`	Get job details
`kubectl get pods -l job-name=NAME`	List pods for job
`kubectl logs POD_NAME`	View pod logs
`kubectl delete job NAME`	Delete job
`kubectl edit job NAME`	Edit job

Job Configuration Keys

Key	Description	Default
`completions`	Total pods to complete	1
`parallelism`	Pods to run in parallel	1
`backoffLimit`	Retry attempts	6
`activeDeadlineSeconds`	Max runtime	None
`ttlSecondsAfterFinished`	Auto-cleanup delay	None
`restartPolicy`	Restart on failure	Always

Conclusion

You've now learned how to run batch processing tasks in Kubernetes using Jobs! This is a powerful feature that allows you to run any task that needs to complete, from database migrations to data processing.

Key Takeaways

Jobs run to completion - They finish and don't restart
Jobs are for batch tasks - Not for continuous services
Use restartPolicy: Never - Prevents infinite loops
Set resource limits - Prevents resource exhaustion
Test commands locally - Saves debugging time
Monitor job status - Know when tasks complete

What You Learned

✅ What Jobs are and when to use them
✅ How to create and configure Jobs
✅ The Job lifecycle and states
✅ How to monitor and verify Jobs
✅ Real-world Job examples
✅ Best practices for production

Next Steps

Now that you've mastered Jobs, consider exploring:

CronJobs - Schedule tasks to run periodically
Parallel Processing - Run multiple pods for large datasets
Workflows - Complex job chains with Argo Workflows
Event-driven Jobs - Trigger jobs based on events
Monitoring - Set up alerts for job failures

Kubernetes Jobs: A Beginner's Guide to Batch Processing

Introduction

What You'll Learn

Table of Contents

What are Kubernetes Jobs?

The Definition

The Problem Jobs Solve

How Jobs Work

Understanding the Problem Jobs Solve

Job vs Pod vs Deployment

When to Use a Job

Step-by-Step Job Creation

The Scenario

Step 1: Generate the YAML

Step 2: Edit the YAML

Step 3: Apply the Job

Step 4: Verify the Job

Step 5: Check Pods

Step 6: View Logs

Step 7: Check Completion

Job Lifecycle and States

Job Lifecycle Diagram

Job States

Pod States

Monitoring and Verification

1. Check Job Status

2. View Pods

3. Check Logs

4. Check Events

5. Complete Verification Script

Advanced Job Configuration

1. Parallel Jobs

2. Job with Failure Handling

3. Job with Resource Limits

4. Auto-Cleanup After Completion

5. Complete Production-Ready Job

Real-World Examples

Example 1: Database Migration

Example 2: Data Processing

Example 3: System Cleanup

Common Issues and Troubleshooting

Issue 1: Pod Not Starting

Issue 2: Command Not Found

Issue 3: Container Exits with Error

Issue 4: Job Never Completes

Issue 5: Too Many Failed Attempts

Best Practices

✅ DO's

❌ DON'Ts

Job Management Commands

Create Job

View Jobs

Update Job

Delete Job

View Pods

Quick Reference Card

Job Commands

Job Configuration Keys

Conclusion

Key Takeaways

What You Learned

Next Steps