Automation Use Cases

4 minute read

Linux’s true power emerges when you use it to automate repetitive tasks. One well-written shell script running on a schedule can save hours of manual work every week.

Cron Jobs for Scheduled Backups

cron is the Linux job scheduler. You define a schedule and a command, and Linux runs it automatically.

Cron Syntax

*  *  *  *  *   command
│  │  │  │  └── Day of week (0-7, Sun=0 or 7)
│  │  │  └───── Month (1-12)
│  │  └──────── Day of month (1-31)
│  └─────────── Hour (0-23)
└──────────────  Minute (0-59)

Backup Script Example

#!/usr/bin/env bash
# /home/ubuntu/scripts/backup-db.sh

set -euo pipefail

BACKUP_DIR="/var/backups/postgres"
DB_NAME="production_db"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="${BACKUP_DIR}/${DB_NAME}_${DATE}.sql.gz"

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Dump and compress the database
pg_dump "$DB_NAME" | gzip > "$BACKUP_FILE"

# Keep only last 7 days of backups
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +7 -delete

echo "Backup created: $BACKUP_FILE"

# Schedule this script to run every night at 2:00 AM
crontab -e

# Add this line:
0 2 * * * /home/ubuntu/scripts/backup-db.sh >> /var/log/backup.log 2>&1

# Verify the cron job is scheduled
crontab -l
# 0 2 * * * /home/ubuntu/scripts/backup-db.sh >> /var/log/backup.log 2>&1

Log Rotation and Archival

Linux servers generate large amounts of log data. Without rotation, logs consume all available disk space.

logrotate Configuration

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily               # Rotate logs every day
    rotate 14           # Keep 14 days of logs
    compress            # Gzip old log files
    delaycompress       # Compress the previous day's log, not today's
    missingok           # Don't error if the log file is missing
    notifempty          # Don't rotate empty log files
    create 0640 ubuntu www-data  # Create new log with these permissions
    postrotate
        # Reload the app so it writes to the new log file
        systemctl reload myapp 2>/dev/null || true
    endscript
}

# Test your logrotate config (dry run)
logrotate --debug /etc/logrotate.d/myapp

# Force rotation immediately (for testing)
logrotate --force /etc/logrotate.d/myapp

# Check logrotate status
cat /var/lib/logrotate/status

System Health Monitoring

Shell scripts can monitor system health and send alerts when thresholds are exceeded.

#!/usr/bin/env bash
# /home/ubuntu/scripts/health-check.sh
# Sends an alert if disk, CPU, or memory usage is too high

set -euo pipefail

ALERT_EMAIL="ops@example.com"
HOSTNAME=$(hostname)

# ── Disk check ────────────────────────────────────────────────
DISK_USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
if [[ $DISK_USAGE -gt 85 ]]; then
    echo "ALERT: Disk usage is ${DISK_USAGE}% on ${HOSTNAME}" \
        | mail -s "Disk Alert: ${HOSTNAME}" "$ALERT_EMAIL"
fi

# ── Memory check ──────────────────────────────────────────────
MEM_TOTAL=$(free -m | awk '/^Mem:/{print $2}')
MEM_USED=$(free -m | awk '/^Mem:/{print $3}')
MEM_PCT=$(( MEM_USED * 100 / MEM_TOTAL ))
if [[ $MEM_PCT -gt 90 ]]; then
    echo "ALERT: Memory usage is ${MEM_PCT}% on ${HOSTNAME}" \
        | mail -s "Memory Alert: ${HOSTNAME}" "$ALERT_EMAIL"
fi

# ── CPU check (1-minute load average vs CPU count) ────────────
CPU_COUNT=$(nproc)
LOAD_AVG=$(awk '{print int($1)}' /proc/loadavg)
if [[ $LOAD_AVG -gt $CPU_COUNT ]]; then
    echo "ALERT: Load average ${LOAD_AVG} exceeds CPU count ${CPU_COUNT} on ${HOSTNAME}" \
        | mail -s "CPU Alert: ${HOSTNAME}" "$ALERT_EMAIL"
fi

# Schedule health check every 5 minutes
crontab -e
# Add:
*/5 * * * * /home/ubuntu/scripts/health-check.sh

Deployment Automation

Shell scripts can automate the entire deployment process — pull new code, build, test, restart services.

#!/usr/bin/env bash
# /home/ubuntu/scripts/deploy.sh
# Zero-downtime deployment for a Node.js application

set -euo pipefail

APP_DIR="/var/www/myapp"
REPO_URL="https://github.com/myorg/myapp.git"
BRANCH="main"

echo "=== Deployment started at $(date) ==="

# 1. Pull latest code
cd "$APP_DIR"
git fetch origin
git checkout "$BRANCH"
git pull origin "$BRANCH"
echo "✓ Code updated to $(git rev-parse --short HEAD)"

# 2. Install dependencies
npm ci --production
echo "✓ Dependencies installed"

# 3. Run database migrations
npm run migrate
echo "✓ Migrations complete"

# 4. Restart the application gracefully
pm2 reload myapp --update-env
echo "✓ Application reloaded"

# 5. Verify the app is responding
sleep 2
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/health)
if [[ "$HTTP_STATUS" == "200" ]]; then
    echo "✓ Health check passed (HTTP 200)"
else
    echo "✗ Health check FAILED (HTTP ${HTTP_STATUS})"
    exit 1
fi

echo "=== Deployment complete at $(date) ==="

Data Pipeline and ETL Scripts

Linux is widely used for ETL (Extract, Transform, Load) workflows that process large data files.

#!/usr/bin/env bash
# /home/ubuntu/scripts/etl-pipeline.sh
# Daily ETL: download CSV from S3, transform, load into PostgreSQL

set -euo pipefail

DATE=$(date +%Y-%m-%d)
S3_BUCKET="s3://company-data-lake"
RAW_FILE="/tmp/sales_${DATE}.csv"
CLEAN_FILE="/tmp/sales_${DATE}_clean.csv"
DB_NAME="analytics"

echo "=== ETL Pipeline: ${DATE} ==="

# 1. Extract: Download today's file from S3
aws s3 cp "${S3_BUCKET}/sales/${DATE}.csv" "$RAW_FILE"
echo "✓ Downloaded: $RAW_FILE ($(wc -l < "$RAW_FILE") rows)"

# 2. Transform: Remove header, clean nulls, filter valid rows
tail -n +2 "$RAW_FILE" \
    | awk -F',' '$3 != "" && $4 > 0 {print}' \
    | sed 's/\r//g' \
    > "$CLEAN_FILE"
echo "✓ Cleaned: $(wc -l < "$CLEAN_FILE") valid rows"

# 3. Load: Import into PostgreSQL
psql -d "$DB_NAME" -c "\copy sales_raw FROM '$CLEAN_FILE' CSV"
echo "✓ Loaded into PostgreSQL"

# 4. Cleanup
rm -f "$RAW_FILE" "$CLEAN_FILE"
echo "=== Pipeline complete ==="

Common Automation Patterns

Pattern Tools Use Case
Scheduled jobs cron, systemd timers Backups, reports, cleanups
Log management logrotate, rsyslog Log rotation, shipping to ELK
Health checks shell scripts + cron Alert on disk/memory/CPU
Deployment git, systemctl, pm2 Application deployments
Data pipelines awk, sed, psql, AWS CLI ETL workflows
Certificate renewal certbot, cron Let’s Encrypt HTTPS certs

Key Takeaways

  • cron is Linux’s built-in job scheduler — five fields define when a command runs
  • logrotate handles log file rotation automatically — configure once, forget it
  • Shell scripts can check disk, memory, and CPU usage and send alerts by email
  • Deployment automation scripts pull code, install deps, run migrations, and restart services
  • Linux powers ETL pipelines with tools like awk, sed, cut, and database CLIs
← Back to Linux in the Cloud Next → Linux vs Windows Server

Updated:

You may also enjoy