Bash Scripting — Using sed and awk in Scripts

2 minute read

Chapter 20 — Using sed and awk in Scripts

Welcome back! In Part 2 of this series, you learned sed and awk as standalone command-line tools. Now it’s time to bring them inside your Bash scripts — where they become incredibly powerful automation building blocks.

This chapter bridges the gap between knowing sed/awk syntax and actually using them in production scripts.

What You’ll Learn

Section Topic Key Skills
20.1 — Capturing Output Capturing sed/awk output $(), mapfile, piping, performance
20.2 — sed in Scripts Using sed in scripts Config replacement, templates, portability
20.3 — awk in Scripts Using awk in scripts -v variables, log parsing, reports
20.4 — Regex in Scripts Bash regex with =~ Validation, BASH_REMATCH, pattern matching
20.5 — Practical Patterns Real-world patterns Log parser, config updater, CSV reports
20.6 — Exercises Practice exercises 5 hands-on challenges

Why Use sed/awk Inside Scripts?

You might wonder — why not just run sed and awk on the command line?

Here’s why embedding them in scripts is a game-changer:

#!/usr/bin/env bash
set -euo pipefail

# 1. Reusable: run the same transformation on any file
# 2. Composable: chain sed/awk with loops, conditions, functions
# 3. Parameterized: pass variables into patterns
# 4. Schedulable: cron jobs, CI/CD pipelines
# 5. Testable: validate output programmatically

Quick Example: Config Updater

#!/usr/bin/env bash
set -euo pipefail

CONFIG="/etc/myapp/app.conf"
KEY="$1"
VALUE="$2"

# Use sed to update a key=value pair in-place
sed -i "s/^${KEY}=.*/${KEY}=${VALUE}/" "$CONFIG"

echo "Updated ${KEY} to ${VALUE} in ${CONFIG}"

Quick Example: Log Summary

#!/usr/bin/env bash
set -euo pipefail

LOG="/var/log/nginx/access.log"

echo "=== Top 5 IPs by Request Count ==="
awk '{print $1}' "$LOG" | sort | uniq -c | sort -rn | head -5

Prerequisites

Before diving in, make sure you’re comfortable with:

  • Bash fundamentals (Chapters 1–19 of this series)
  • sed basics — substitutions, deletions, address ranges (Part 2, sed chapters)
  • awk basics — fields, patterns, actions, BEGIN/END blocks (Part 2, awk chapters)
  • Regular expressions — character classes, quantifiers, groups (Part 2, regex chapter)

💡 Tip: If you need a refresher on sed or awk syntax, revisit the Part 2 chapters before continuing. This chapter assumes you know the tools — we focus on integration patterns.

How This Chapter Is Organized

Each section builds on the previous one:

  1. Capturing Output — The foundation: how to get sed/awk results into variables and arrays
  2. sed in Scripts — Common sed patterns you’ll use in automation
  3. awk in Scripts — Powerful awk patterns for parsing and reporting
  4. Regex in Scripts — Bash-native regex with [[ =~ ]] for validation
  5. Practical Patterns — Complete, real-world script examples
  6. Exercises — Test your skills with progressively harder challenges

Let’s Get Started

Head to 20.1 — Capturing sed/awk Output in Scripts to begin!

Chapter 20 of the Linux Bash Scripting series on timesofcloud.com

Updated:

You may also enjoy