Linux Basic — xargs & GNU Parallel

xargs & GNU Parallel

The fastest way to speed up a loop: run it in parallel. Both xargs and GNU parallel turn a list of inputs into concurrent command executions.

---

xargs Basics

xargs reads lines from stdin and passes them as arguments to a command:

ls *.txt | xargs rm
# equivalent to: rm file1.txt file2.txt file3.txt ...

---

Parallelise with xargs -P

# serial — one at a time
find . -name "*.jpg" | xargs -I {} convert {} {}.png

# parallel — 4 at a time
find . -name "*.jpg" | xargs -P 4 -I {} convert {} {}.png

-P N runs up to N commands concurrently.

---

xargs Quick Reference

Flag	Meaning
-P N	run N jobs in parallel
-I {}	replace {} with each input line
-n N	pass N args per invocation
-0	null-delimited input (pairs with find -print0)
-r	don’t run if input is empty

---

Safe File Handling with -print0 / -0

Filenames with spaces or newlines break plain xargs:

# UNSAFE
find . -name "*.log" | xargs rm

# SAFE
find . -name "*.log" -print0 | xargs -0 rm

-print0 separates with \0 (null byte), which no filename can contain.

---

GNU Parallel

GNU Parallel is xargs on steroids — better syntax, progress bars, retry logic, and remote execution.

Install

# macOS
brew install parallel

# Ubuntu/Debian
apt install parallel

Basic Usage

# convert images in parallel
find . -name "*.jpg" | parallel convert {} {.}.png

# {.} is the basename without extension

Parallel with function

process() {
    echo "processing $1"
    sleep 1
}
export -f process

find . -name "*.txt" | parallel process

---

Parallel Options

Flag	Meaning
-j N	run N jobs (default: num cores)
--bar	show progress bar
--eta	estimate time remaining
--dry-run	print commands without running
--retry-failed	re-run failed jobs

parallel -j 8 --bar process ::: file1 file2 file3 file4

---

When Parallelism Helps

Workload	Parallelism Gain
CPU-bound (compression, encoding)	~linear with cores
I/O-bound (network requests, file reads)	high (limited by I/O, not CPU)
Serial dependencies	none — can’t parallelise

If each job is < 0.1 seconds, the fork overhead may exceed the gain.

---

Benchmark: Serial vs Parallel

# serial (100 files × 1s each = 100s)
time for f in *.jpg; do convert "$f" "${f%.jpg}.png"; done

# parallel -j 8 (100 / 8 ≈ 12.5s + overhead ≈ 15s)
time find . -name "*.jpg" | parallel -j 8 convert {} {.}.png

6–8× speedup on an 8-core machine.

---

Caveat: Resource Limits

Running 100 jobs in parallel can exhaust:

• File descriptors
• Memory (if each job uses 500MB)
• Network connections

Start with -P = num cores, then tune.

---

Next → Practice Exercises