Linux Basic — sed & awk Practical Patterns

4 minute read

Practical Patterns

Ten recipes you’ll reach for again and again. Each shows the goal, the command, and (where it matters) the why.

1. Clean a Config File (Strip Comments & Blank Lines)

sed -E '/^[[:space:]]*#/d; /^[[:space:]]*$/d' config.txt

d deletes; the two address regexes match (a) lines starting with optional whitespace then #, and (b) lines that are entirely whitespace.

To modify in place with backup:

sed -i.bak -E '/^[[:space:]]*#/d; /^[[:space:]]*$/d' config.txt

2. Replace a Setting in a Config File

# Set or update key=value
key="timeout"
new="60"

if grep -q "^${key}=" config.txt; then
    sed -i.bak "s/^${key}=.*/${key}=${new}/" config.txt
else
    echo "${key}=${new}" >> config.txt
fi

Or with awk in one pass (creates the line if missing):

awk -v k="$key" -v v="$new" '
BEGIN { FS=OFS="=" }
$1 == k { $2 = v; found=1 }
{ print }
END { if (!found) print k OFS v }
' config.txt > config.tmp && mv config.tmp config.txt

3. Top N HTTP Status Codes

awk '{ c[$9]++ } END { for (s in c) print c[s], s }' access.log \
    | sort -rn \
    | head -10

$9 is the status field in a default Combined Log Format. Adjust the column number for your log.

4. Per-IP Request Count + Bandwidth

awk '{ count[$1]++; bytes[$1] += $10 } END {
    for (ip in count) printf "%-16s %6d req %12d B\n", ip, count[ip], bytes[ip]
}' access.log | sort -k2 -rn | head -20

Two accumulators per key — the awk way.

5. Extract a Field From a Path

Native is fastest:

path="/var/log/nginx/access.log"
echo "${path##*/}"           # access.log
echo "${path%/*}"            # /var/log/nginx

But if data lives in a file and you need it inside a pipeline:

echo "$path" | awk -F/ '{ print $NF }'
echo "$path" | sed 's|.*/||'

In a hot loop, use native. In a one-shot pipeline, use whatever reads best.

6. CSV Sum + Average

# col 3 = amount
awk -F, '
NR == 1 { next }                  # skip header
        { sum += $3; n++ }
END     { printf "rows=%d sum=%.2f avg=%.2f\n", n, sum, sum/n }
' sales.csv

For multi-column averages:

awk -F, '
NR == 1 { next }
{ for (i=2; i<=NF; i++) { sum[i]+=$i; cnt[i]++ } }
END { for (i in sum) printf "col%d avg=%.2f\n", i, sum[i]/cnt[i] }
' data.csv

7. Filter Log by Time Range

# /var/log/syslog timestamps look like:  Jun 17 09:12:34
start="Jun 17 09:00:00"
end="Jun 17 10:00:00"

awk -v s="$start" -v e="$end" '
{
    ts = $1 " " $2 " " $3
    if (ts >= s && ts <= e) print
}
' /var/log/syslog

awk does lexical comparison — it works as long as your timestamps sort lexically (zero-padded, year omitted means stable within the year).

8. Inject a Block into a File at a Marker

# Insert <new block> after a line matching "# CONFIG MARKER"
sed -i '/# CONFIG MARKER/r additional.conf' main.conf

r FILE reads the named file in after a matching address.

Or with awk:

awk '/# CONFIG MARKER/ { print; while ((getline line < "additional.conf") > 0) print line; next } { print }' \
    main.conf > main.new && mv main.new main.conf

9. Find Lines Present in File A but Not in File B

awk 'NR==FNR { seen[$0]; next } !($0 in seen)' b.txt a.txt

The NR==FNR idiom loads b.txt into a set, then for each line of a.txt prints it only if it wasn’t in b.txt. Much faster than grep -vxF -f.

10. Extract Key Values from a JSON-Like Line (No jq)

For real JSON, use jq. For ad-hoc grepping when jq isn’t installed:

line='{"user":"alice","status":200,"ms":42}'
echo "$line" | sed -E 's/.*"user":"([^"]+)".*/\1/'
# alice

Or with awk:

echo "$line" | awk -F'"' '{ for (i=1; i<=NF; i++) if ($i=="user") print $(i+2) }'

These are brittle by design — they assume well-formed input. For anything user-facing, install jq (covered in Using jq in Chapter 18).

Bonus — Stream Editing With Watch

For long-running tail-and-transform:

tail -F app.log | awk '/ERROR/ { print strftime("%H:%M:%S"), $0; fflush() }'

fflush() flushes after every line — important because awk’s stdout is block-buffered when piped, which can delay output by tens of seconds otherwise. (GNU awk only.)

For mawk / BSD awk, use the line-buffering wrapper:

stdbuf -oL tail -F app.log | awk '/ERROR/ { print }' | tee err.log

The Decision Recap

Situation Tool
String already in a Bash var, simple transform parameter expansion
String already in a Bash var, real regex (no replace) [[ … =~ … ]] + BASH_REMATCH
Stream/file, line-by-line substitution sed -E
Stream/file, column data / aggregation / state awk
JSON / YAML / TOML jq / yq / tq (next chapter or Chapter 18)

Common Mistakes

❌ Wrong ✅ Right Issue
cat file \| awk … awk … file useless use of cat
grep pattern \| awk '…' awk '/pattern/ { … }' awk filters by itself
Many seds piped one sed -e … -e … each pipe is a fork
Parsing JSON with sed/awk in production use jq regex JSON is brittle
awk -v x=$x (unquoted) awk -v x="$x" spaces break it

Next → Practice Exercises

Updated:

You may also enjoy