Linux Basic — Appending to Files in Bash

4 minute read

Appending to Files

Appending sounds simple — slap >> on the end and you are done. In practice you will hit three traps: appending a hundred lines in a loop (slow), two scripts writing the same log file at once (corruption), and the missing trailing newline that makes downstream tools misbehave.

The >> Operator

echo "first run"  >> app.log
echo "second run" >> app.log
cat app.log
# first run
# second run

>> always:

  • Creates the file if it doesn’t exist
  • Seeks to end before each write
  • Never truncates

That last property is the whole reason it exists: > is destructive, >> is additive.

Avoid Appending in Tight Loops

Each >> is a separate open/write/close syscall set. In a loop of thousands of lines, that overhead adds up fast.

# ❌ One redirect per iteration — slow
for i in {1..10000}; do
    echo "line $i" >> big.log
done

# ✅ One redirect for the whole loop — fast
for i in {1..10000}; do
    echo "line $i"
done >> big.log

Same output. The second form opens big.log once, writes the entire loop’s output to that file descriptor, then closes it.

The difference on a typical laptop:

time (for i in {1..10000}; do echo "$i" >> /tmp/a; done; rm /tmp/a)
# real    0m1.234s

time (for i in {1..10000}; do echo "$i"; done >> /tmp/a; rm /tmp/a)
# real    0m0.087s

~14× faster with no change in behaviour.

Grouping Commands Before Appending

{
    echo "===== Build run $(date) ====="
    echo "Branch: $(git rev-parse --abbrev-ref HEAD)"
    echo "Commit: $(git rev-parse --short HEAD)"
    echo
} >> build.log

{ cmd1; cmd2; } runs in the current shell (unlike ( ... ) which is a subshell). Group, redirect once, done.

You can also pipe a group:

{ echo "header"; cat data.txt; } >> archive.log

Concurrent Writers — Use flock

If two processes append to the same file at exactly the same instant, their writes can interleave inside a single line. POSIX guarantees only that writes under a certain size are atomic — but the threshold depends on the kernel and filesystem.

# Two scripts running every minute, both appending — corruption risk
script_a >> shared.log &
script_b >> shared.log &

The fix is advisory file locking with flock(1):

(
    flock -x 200          # take exclusive lock on FD 200; wait if needed
    echo "[$(date)] $message" >> shared.log
) 200>> shared.log        # FD 200 is opened to the file

Or the one-line variant:

flock shared.log -c 'echo "[$(date)] $msg" >> shared.log'

flock blocks until the lock is free. To fail instead of waiting:

flock -n shared.log -c 'echo "$msg" >> shared.log' \
    || echo "another writer is busy" >&2
Flag Meaning
-x / -e exclusive lock (writer)
-s shared lock (reader)
-n non-blocking — fail immediately if locked
-w SECS wait up to N seconds, then fail

Log Rotation Pattern

You usually don’t roll your own log rotation — logrotate exists. But the manual idiom is good to know:

# Roll the current log, start a fresh one
mv app.log "app.log.$(date +%F_%H%M)"
:> app.log    # create empty file (':' is a no-op; combined with > truncates/creates)

# Then keep appending as before
echo "fresh start" >> app.log

Compress old logs:

gzip app.log.2025-04-01_1200
ls
# app.log
# app.log.2025-04-01_1200.gz

Delete logs older than 30 days:

find . -name "app.log.*" -mtime +30 -delete

Real systems should use logrotate (covered in Chapter 19).

The Trailing-Newline Trap

POSIX text files are supposed to end with a newline. Many tools — wc -l, awk, grep -c, tail, even some loops — count records by newlines, so a missing trailing newline can cost you the last record.

# A file ending without a newline
printf 'line1\nline2' > no-nl.txt
wc -l no-nl.txt
# 1 no-nl.txt     ← reports 1 even though there are 2 lines

# Append safely — make sure each entry ends with \n
printf 'line3\n' >> no-nl.txt
wc -l no-nl.txt
# 3 no-nl.txt     ← but now line2 and line3 are on the same line!

Two safe patterns:

# Always use printf with explicit \n
printf '%s\n' "$message" >> app.log

# Or check & fix before appending
[ -s file ] && [ "$(tail -c1 file)" != $'\n' ] && echo >> file
echo "$message" >> file

Append From Standard Input

# Append the output of a pipeline
date | tee -a app.log
# Sun  1 Jun 2026 00:00:00 +0530
cat app.log
# Sun  1 Jun 2026 00:00:00 +0530

# Append with sudo to a root-owned file (see next page)
echo "127.0.0.1 myhost" | sudo tee -a /etc/hosts > /dev/null

>> and sudo together don’t work the way you might expect — sudo only elevates the command, not the redirection. tee -a is the standard workaround. The next page covers tee in full.

Common Mistakes

❌ Wrong ✅ Right Why
for i in ...; do echo "$i" >> f; done for i in ...; do echo "$i"; done >> f one fopen per iter is slow
Two scripts >> log concurrently flock around each writer interleaved output corruption
sudo echo x >> /etc/hosts echo x \| sudo tee -a /etc/hosts >/dev/null sudo doesn’t elevate the redirect
printf 'x' >> file printf 'x\n' >> file missing newline merges with next line
cmd > /tmp/out && cat /tmp/out >> log cmd \| tee -a log (or cmd >> log) needless tempfile

Next → Writing with tee

Updated:

You may also enjoy