GCP for DevOps — Storage for DevOps

Storage for DevOps

Cloud Storage is the backbone of many DevOps workflows. This page covers practical patterns for Terraform state, build artifacts, log archival, and backup automation.

Terraform State Backend

Store Terraform state in a versioned GCS bucket with locking via Cloud Storage’s built-in generation-based locking.

# Create the state bucket
$ gcloud storage buckets create gs://myco-terraform-state \
    --location=us-central1 \
    --default-storage-class=STANDARD \
    --uniform-bucket-level-access

# Enable versioning (critical for state recovery)
$ gcloud storage buckets update gs://myco-terraform-state \
    --versioning

Terraform backend configuration:

terraform {
  backend "gcs" {
    bucket = "myco-terraform-state"
    prefix = "environments/production"
  }
}

┌──────────────────────────────────────────────────┐
│             Terraform State Flow                  │
├──────────────────────────────────────────────────┤
│                                                   │
│  terraform plan/apply                             │
│        │                                          │
│        ▼                                          │
│  ┌───────────┐    lock     ┌──────────────────┐  │
│  │  Terraform ├────────────►  GCS Bucket       │  │
│  │  CLI       │◄────────────  (versioned)      │  │
│  └───────────┘  read/write └──────────────────┘  │
│                                                   │
│  Version History:                                 │
│  ├── #generation3  (current)                      │
│  ├── #generation2  (previous)                     │
│  └── #generation1  (oldest)                       │
└──────────────────────────────────────────────────┘

Build Artifacts Storage

Store CI/CD build outputs for deployment or debugging.

# Create an artifacts bucket
$ gcloud storage buckets create gs://myco-build-artifacts \
    --location=us-central1

# Upload build artifacts with metadata
$ gcloud storage cp build/app-v1.2.3.tar.gz \
    gs://myco-build-artifacts/releases/app-v1.2.3.tar.gz \
    --custom-metadata=commit-sha=abc1234,branch=main,build-id=42

# List artifacts with metadata
$ gcloud storage objects describe \
    gs://myco-build-artifacts/releases/app-v1.2.3.tar.gz \
    --format="yaml(metadata)"

# Cloud Build — save artifacts in cloudbuild.yaml
# artifacts:
#   objects:
#     location: gs://myco-build-artifacts/builds/$BUILD_ID/
#     paths: ["build/*.tar.gz"]

Log Archival Pattern

Export logs from Cloud Logging to Cloud Storage for long-term retention.

# Create a log sink that exports to GCS
$ gcloud logging sinks create audit-log-archive \
    storage.googleapis.com/myco-audit-logs \
    --log-filter='logName="projects/my-project/logs/cloudaudit.googleapis.com%2Factivity"'

# Grant the sink's service account write access
# (get the writer identity from the sink)
$ WRITER=$(gcloud logging sinks describe audit-log-archive \
    --format="value(writerIdentity)")

$ gcloud storage buckets add-iam-policy-binding gs://myco-audit-logs \
    --member="$WRITER" \
    --role="roles/storage.objectCreator"

Backup Automation with gsutil / gcloud

# Sync a directory to GCS (incremental backup)
$ gcloud storage rsync /var/backups/db/ gs://myco-backups/database/ \
    --recursive \
    --delete-unmatched-destination-objects

# Parallel upload large files
$ gcloud storage cp large-backup.tar.gz gs://myco-backups/ \
    --parallelism-count=8

Cross-Region Replication

Use dual-region or multi-region buckets for disaster recovery.

# Create a dual-region bucket (us-central1 + us-east1)
$ gcloud storage buckets create gs://myco-dr-backups \
    --location=US \
    --default-storage-class=STANDARD \
    --placement=us-central1,us-east1

# Turbo replication (15-minute RPO, additional cost)
$ gcloud storage buckets update gs://myco-dr-backups \
    --rpo=ASYNC_TURBO

Storage Notifications

Trigger workflows when objects are created or deleted.

# Create a Pub/Sub notification
$ gcloud storage buckets notifications create \
    gs://myco-build-artifacts \
    --topic=artifact-events \
    --event-types=OBJECT_FINALIZE

# List notifications on a bucket
$ gcloud storage buckets notifications list gs://myco-build-artifacts

DevOps Storage Patterns Summary

Use Case	Bucket Config	Key Settings
Terraform state	Standard, single-region	Versioning, lifecycle (keep N versions)
Build artifacts	Standard, single-region	30-day lifecycle delete, metadata
Log archival	Nearline → Archive	Lifecycle tiering, retention lock
Database backups	Dual-region or multi-region	Versioning, turbo replication
Static assets	Multi-region	CDN integration, public read

Common Mistakes

Mistake	Why It’s Wrong	Fix
State bucket without versioning	Can’t recover corrupt state	Always enable versioning
Single-region for DR backups	Region outage loses backups	Use dual-region or multi-region
No lifecycle on artifact buckets	Costs grow from old artifacts	Add age-based deletion
Missing metadata on artifacts	Can’t trace build origin	Add commit SHA, branch, build ID
Not granting sink writer access	Log export silently fails	Check and grant writer identity

Quick Reference

Pattern	Key Command
State bucket	gcloud storage buckets create gs://B --versioning
Artifact upload	gcloud storage cp FILE gs://B/ --custom-metadata=k=v
Log sink	gcloud logging sinks create NAME storage.googleapis.com/B
Backup sync	gcloud storage rsync /local/ gs://B/ --recursive
Notifications	gcloud storage buckets notifications create gs://B --topic=T

Next → Persistent Disks