Linux Basic — awk in Scripts

4 minute read

awk in Scripts

If your data has columns, awk is the right tool. It splits each line into fields automatically and gives you a tiny programming language for filtering, transforming, and summing them.

This page covers the patterns that show up in real scripts — not the full awk language.

Field Basics

By default, awk splits on runs of whitespace:

echo "alice 42 admin" | awk '{ print $2 }'
# 42

echo "alice 42 admin" | awk '{ print $1, $3 }'
# alice admin

Variables:

Var Meaning
$0 the whole line
$N field N (1-based)
$NF last field
NF number of fields on this line
NR line number (1-based)
FS input field separator
OFS output field separator (default: space) 
echo "/var/log/nginx/access.log" | awk -F/ '{ print $NF }'
# access.log

-F sets FS. For comma-separated:

echo "alice,42,admin" | awk -F, '{ print $1, $3 }'
# alice admin

For tab-separated, use -F'\t'.

Conditional Selection — pattern { action }

# Print lines longer than 80 chars
awk 'length($0) > 80' file.txt

# Print only lines where column 3 == "admin"
awk -F, '$3 == "admin" { print $1 }' users.csv

# Print lines matching a regex
awk '/ERROR/ { print }' app.log

# Lines NOT matching
awk '!/DEBUG/' app.log

A naked pattern with no action means “print the line.” A naked {action} with no pattern means “do this for every line.”

BEGIN and END — Setup and Aggregation

# Sum column 3
awk -F, '{ total += $3 } END { print total }' sales.csv

# Average response time from access log
awk '{ sum += $NF; n++ } END { printf "avg %.2fms\n", sum/n }' access.log

# Header + per-record + footer
awk -F, '
BEGIN { print "User,Count" }
      { count[$1]++ }
END   { for (u in count) printf "%s,%d\n", u, count[u] }
' actions.csv

BEGIN { ... } runs once before any input, END { ... } runs once after all input.

Associative Arrays — The Killer Feature

Awk’s arrays are hash tables. That makes counting and grouping trivial:

# Count occurrences of each HTTP status code
awk '{ counts[$9]++ } END { for (c in counts) print c, counts[c] }' access.log
# 200 12453
# 301   234
# 404    87
# 500     5

Sort the output by piping to sort:

awk '{ c[$9]++ } END { for (k in c) print c[k], k }' access.log | sort -rn

Passing Bash Variables — -v

The right way:

threshold=100

awk -v t="$threshold" '$3 > t { print $1 }' data.txt

-v defines an awk variable from outside. Each -v is one var:

awk -v min=10 -v max=100 '$3 >= min && $3 <= max' data.txt

The wrong way (don’t interpolate variables into the program):

# ❌ Quoting hell and a code-injection risk
awk "{ if (\$3 > $threshold) print \$1 }" data.txt

Always pass through -v.

Printing — print vs printf

# print: simple, adds OFS between args and ORS (\n) at end
awk '{ print $1, $2 }' file.txt

# printf: full format control, NO newline added
awk '{ printf "%-15s %5d\n", $1, $2 }' file.txt

Common format specifiers: %s string, %d integer, %f float, %-10s left-pad to 10, %5d right-pad to 5, %.2f 2 decimals.

Multi-Pattern Programs

awk '
/^GET/  { gets++ }
/^POST/ { posts++ }
END     { print "GET:", gets; print "POST:", posts }
' access.log

For longer scripts, store in a file and use -f:

awk -f process.awk data.csv

A Real Example — Top Talkers

# Top 10 source IPs from an access log
awk '{ ip[$1]++ } END { for (k in ip) print ip[k], k }' access.log \
    | sort -rn \
    | head -10

Three commands, no for loop in Bash. Native awk arrays handle the counting; sort -rn orders by count desc; head -10 trims.

A Real Example — Per-User Disk Usage Report

# /etc/passwd → /home, sum sizes
du -s /home/*/ 2>/dev/null | awk '
{
    n = split($2, parts, "/")
    user = parts[n-1]            # /home/<user>/  → user is second-to-last
    size_mb = $1 / 1024
    if (size_mb > 100) {
        printf "%-15s %8.1f MB\n", user, size_mb
    }
}
'

split(string, array, sep) returns the field count and populates array 1-indexed. This is the awk way to do “split on slash.”

Multi-Char Field Separators (FS Is a Regex)

# Split on any of : / -
echo "a:b/c-d" | awk -F'[:/-]' '{ print $1, $2, $3, $4 }'
# a b c d

For literal multi-char separators (<->), use a regex too:

echo "a<->b<->c" | awk -F'<->' '{ print $2 }'
# b

(That’s full ERE, so quote any regex meta-chars literally.)

awk + getline (Read Another File)

# Join: lookup a name from /etc/passwd for each UID in input
awk -F: 'NR==FNR { name[$3]=$1; next } { print $1, name[$1] }' /etc/passwd input.txt

NR==FNR is the standard idiom: “while reading the first file (NR == FNR), build a map; then for subsequent files, do the lookup.”

Common Mistakes

❌ Wrong ✅ Right Issue
awk "{print \$$col}" (interpolation) awk -v c="$col" '{print $c}' quoting + injection
awk '{print $1}' on CSV with quoted fields use a CSV parser or csvkit naive splitter breaks on "a,b",c
for line in $(cat file); do awk … awk 'pattern' file move the loop into awk
print $NF + 1 when $NF is a string awk auto-coerces, but verify numeric ops on strings give 0
Forgetting END for sums put aggregation in END otherwise nothing prints

Next → Regex with =~

Updated:

You may also enjoy