Linux Basic — awk Regex Patterns

awk Regex Patterns

awk is regex-driven by design. Every rule is pattern { action } where the pattern is usually a regex. It uses ERE natively — no -E needed.

awk '/^ERROR/ { print $0 }' app.log

That one-liner is grep '^ERROR' — but awk keeps going where grep stops: column-aware fields, math, conditionals, and accumulators in a single tool.

---

Pattern Forms

Pattern	Runs the action when…
/regex/	the whole line matches regex
$N ~ /regex/	field N matches regex
$N !~ /regex/	field N does not match
condition && /regex/	both true
NR == 1	first line
BEGIN { ... }	before any input
END { ... }	after all input

# 5th field is a number
awk '$5 ~ /^[0-9]+$/ { print }'

# warnings AND fatal lines
awk '/WARN/ || /FATAL/ { print NR, $0 }'

# skip comments and blanks
awk '!/^[[:space:]]*(#|$)/' config

---

Field-Level Matching

awk splits each line into $1, $2, … on whitespace by default (or FS):

# users whose shell is /bin/bash (field 7 of /etc/passwd)
awk -F: '$7 ~ /bash$/ {print $1}' /etc/passwd

# IPs (field 1) hitting /api/
awk '$7 ~ /^\/api\// {print $1}' access.log | sort | uniq -c

-F: sets the field separator to :. Use -F'\t' for tabs.

---

match() and RSTART / RLENGTH

match(string, regex) returns the position of the first match (0 if none) and sets RSTART and RLENGTH:

awk '{
    if (match($0, /[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)) {
        ip = substr($0, RSTART, RLENGTH)
        print ip
    }
}' access.log

Gawk also offers match(str, re, arr) which fills arr with capture groups (gawk extension).

---

gsub() / sub() — Substitute Inside awk

# strip everything after #
awk '{ sub(/#.*$/, ""); print }'

# normalise multiple spaces
awk '{ gsub(/[[:space:]]+/, " "); print }'

# replace inside a single field, then print all
awk '{ gsub(/foo/, "bar", $3); print }'

sub replaces the first match; gsub replaces all. Both modify in place and return the count of replacements.

---

Practical One-Liners

# count lines matching a pattern (per file)
awk '/ERROR/ {n++} END{print FILENAME, n+0}' *.log

# sum the 5th field where 2nd field is "in"
awk '$2 == "in" {s += $5} END {print s}' transactions

# print line N (cheap sed -n)
awk 'NR==42' file

# print every 10th line
awk 'NR % 10 == 0' file

# remove duplicate adjacent lines (no sort needed)
awk 'prev != $0; { prev = $0 }' file

# CSV: print row where column 3 looks like an email
awk -F, '$3 ~ /@/ {print $1, $3}' contacts.csv

---

BEGIN / END for Accumulators

awk '
    BEGIN { FS=":"; print "User,UID" }
    $3 >= 1000 { print $1 "," $3 }
    END   { print "---" NR " entries scanned" }
' /etc/passwd

BEGIN runs once before input; END runs once after. Both can do real work — no if NR==1 hacks needed.

---

awk vs grep vs sed

Need	Best tool
“Does any line match?” — yes/no	grep -q
Filter lines by regex, no transform	grep
Substitute pattern → replacement	sed
Filter on a specific field, do math, accumulate	awk

When the question involves columns, arithmetic, or “sum / count / group by”, reach for awk first.

---

Pitfalls

Pitfall	Fix
Slash in regex needs escaping: /\/api\//	Or use string match: $1 ~ "/api/"
Forgetting -F for non-whitespace separators	awk -F: … for /etc/passwd
print adds newline; printf doesn’t	Use printf "%s\n" for control
Reading huge files into arrays	OK if you have RAM; otherwise stream

---

Next → Practice Exercises