Linux Basic — sed & awk in Scripts

sed & awk in Scripts

Bash gives you fast native string operations (last chapter), but the moment you need real regex, multi-line transforms, or column-oriented data, the right tool is sed or awk. This chapter is about using them inside scripts — capturing their output safely, knowing when each one shines, and weaving them into a robust Bash workflow.

We assume you’ve already met the basics of sed and awk at the command line. Here we focus on the scripting angle.

---

What You Will Learn

By the end of this chapter you will be able to:

• Capture command output into variables with $(...) — and know its pitfalls
• Use sed from a script for substitutions, deletions, and in-place edits (with the macOS/BSD vs GNU gotcha)
• Use awk for field extraction, aggregation, and conditional filtering
• Use Bash’s native [[ string =~ regex ]] operator with BASH_REMATCH
• Recognise the practical patterns that come up in 90% of real scripts
• Decide quickly: parameter expansion vs sed vs awk

---

Topics in This Chapter

#	Topic	Description
1	Capturing Output	$(...) vs backticks, trailing-newline trap, mapfile
2	sed in Scripts	substitution, in-place edit, GNU vs BSD -i
3	awk in Scripts	columns, conditions, aggregation, -v
4	Regex with =~	native ERE matching and BASH_REMATCH
5	Practical Patterns	10 recipes you’ll reuse forever
6	Practice Exercises	Hands-on tasks

---

The Decision Tree

When manipulating text in a script, ask in this order:

1. Is the data already in a Bash variable? → try parameter expansion first (${var/x/y}, ${var##*/})
2. Do I need real regex? → [[ "$s" =~ regex ]] then BASH_REMATCH
3. Single-line substitutions in a stream/file? → sed
4. Column-oriented data, aggregation, conditional output? → awk

Reach for sed/awk when Bash genuinely can’t do it cleanly. In hot loops, do it natively. In one-shot pipelines, use whatever reads best.

---

Next → Capturing Output