ELK Stack — Log Retention in Kubernetes

5 minute read

Log Storage Considerations in Kubernetes

Kubernetes clusters generate an enormous volume of logs. A 50-node production cluster running microservices can easily produce hundreds of gigabytes per day. Without a clear retention strategy, storage costs spiral out of control and Elasticsearch performance degrades.

This page covers how to manage Elasticsearch storage on Kubernetes — from persistent volumes and storage classes to ILM policies and the hot-warm-cold architecture.

PersistentVolumes for Elasticsearch Data

When running Elasticsearch on Kubernetes (via ECK or StatefulSets), each data node needs a PersistentVolumeClaim (PVC) for durable storage.

volumeClaimTemplates:
  - metadata:
      name: elasticsearch-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 100Gi

Key Points

Always use PVCs — never use emptyDir for Elasticsearch data in production.
Use ReadWriteOnce — each ES node needs its own volume.
Size volumes for your retention period plus a 20% buffer.
Use volume expansion if your StorageClass supports it (allowVolumeExpansion: true).

StorageClass Selection

The StorageClass determines the type of disk backing your Elasticsearch volumes. Choose based on the data tier:

Tier	StorageClass	Disk Type	Use Case
Hot	`fast-ssd` / `gp3`	SSD	Current day’s logs, frequent queries
Warm	`standard` / `st1`	HDD or slow SSD	7-30 day old logs, occasional queries
Cold	`cold-hdd` / `sc1`	Cheapest HDD	30-90 day old logs, rare access

Example StorageClass for AWS EBS:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cold-hdd
provisioner: ebs.csi.aws.com
parameters:
  type: st1
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

ILM Policies for Kubernetes Logs

Index Lifecycle Management (ILM) automates the transition and deletion of indices based on age and size.

Complete ILM Policy for K8s Logs

PUT _ilm/policy/k8s-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "30gb",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "14d",
        "actions": {
          "allocate": {
            "require": {
              "data": "cold"
            }
          },
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Apply ILM to an Index Template

PUT _index_template/k8s-logs-template
{
  "index_patterns": ["k8s-logs-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "k8s-logs-policy",
      "index.lifecycle.rollover_alias": "k8s-logs",
      "number_of_shards": 2,
      "number_of_replicas": 1
    }
  }
}

Bootstrap the First Index

PUT k8s-logs-000001
{
  "aliases": {
    "k8s-logs": {
      "is_write_index": true
    }
  }
}

Now configure Filebeat or Fluent Bit to write to the k8s-logs alias instead of a date-based index name.

Snapshot Lifecycle Management (SLM)

Before deleting old indices, snapshot them to object storage (S3, GCS, Azure Blob) for compliance and long-term archival.

Register a Snapshot Repository

PUT _snapshot/k8s-backup
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-backups",
    "region": "us-east-1",
    "base_path": "k8s-cluster"
  }
}

Create an SLM Policy

PUT _slm/policy/nightly-k8s-snapshots
{
  "schedule": "0 30 2 * * ?",
  "name": "<k8s-snap-{now/d}>",
  "repository": "k8s-backup",
  "config": {
    "indices": ["k8s-logs-*"],
    "ignore_unavailable": true,
    "include_global_state": false
  },
  "retention": {
    "expire_after": "90d",
    "min_count": 5,
    "max_count": 50
  }
}

Curator (Legacy Context)

Before ILM was built into Elasticsearch (version 6.6+), Curator was the standard tool for managing index lifecycle. Curator is a Python CLI that runs as a Kubernetes CronJob.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: es-curator
  namespace: elastic-system
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: curator
              image: bitnami/elasticsearch-curator:5.8
              args: ["--config", "/etc/curator/config.yml", "/etc/curator/actions.yml"]
              volumeMounts:
                - name: config
                  mountPath: /etc/curator
          restartPolicy: OnFailure
          volumes:
            - name: config
              configMap:
                name: curator-config

Recommendation: Use ILM for Elasticsearch 7.x and later. Curator is only needed for legacy clusters running 6.x or earlier.

Retention Policies Based on Namespaces

Different namespaces often have different retention requirements. Production logs may need 90 days; development logs may only need 3 days.

Strategy — Separate Index per Namespace

Configure Filebeat (or Fluent Bit) to route logs to namespace-specific indices:

# Filebeat ConfigMap
filebeat.inputs:
  - type: container
    paths:
      - /var/log/containers/*.log
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  index: "k8s-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}"

setup.ilm.enabled: false
setup.template.name: "k8s"
setup.template.pattern: "k8s-*"

Then create separate ILM policies:

PUT _ilm/policy/k8s-production-policy
{
  "policy": {
    "phases": {
      "hot":    { "min_age": "0ms", "actions": { "rollover": { "max_age": "1d" } } },
      "warm":   { "min_age": "7d",  "actions": { "forcemerge": { "max_num_segments": 1 } } },
      "cold":   { "min_age": "30d", "actions": {} },
      "delete": { "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}

PUT _ilm/policy/k8s-development-policy
{
  "policy": {
    "phases": {
      "hot":    { "min_age": "0ms", "actions": { "rollover": { "max_age": "1d" } } },
      "delete": { "min_age": "3d",  "actions": { "delete": {} } }
    }
  }
}

Cost Optimization — Hot-Warm-Cold on Kubernetes

Implement the hot-warm-cold architecture by assigning different ECK node sets to different StorageClasses and node pools.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 8.13.0
  nodeSets:
    - name: hot
      count: 3
      config:
        node.roles: ["data_hot", "ingest", "master"]
        node.attr.data: hot
      podTemplate:
        spec:
          nodeSelector:
            node-pool: hot
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            storageClassName: fast-ssd
            resources:
              requests:
                storage: 100Gi

    - name: warm
      count: 2
      config:
        node.roles: ["data_warm"]
        node.attr.data: warm
      podTemplate:
        spec:
          nodeSelector:
            node-pool: warm
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            storageClassName: standard
            resources:
              requests:
                storage: 500Gi

    - name: cold
      count: 1
      config:
        node.roles: ["data_cold"]
        node.attr.data: cold
      podTemplate:
        spec:
          nodeSelector:
            node-pool: cold
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            storageClassName: cold-hdd
            resources:
              requests:
                storage: 1Ti

Index Naming Strategies for K8s Logs

Strategy	Index Pattern	Use Case
Date-based	`k8s-logs-2026.04.01`	Simple, easy to delete by date
Namespace-based	`k8s-production-2026.04.01`	Different retention per namespace
App-based	`k8s-frontend-2026.04.01`	Per-service dashboards
Cluster-based	`cluster-east-k8s-2026.04.01`	Multi-cluster environments
Combined	`k8s-prod-frontend-2026.04.01`	Fine-grained control

Best practice: Use k8s-<namespace>-<date> as the default pattern. It balances granularity with manageability.

Storage Planning Guide

Use this formula to estimate Elasticsearch storage requirements:

Daily log volume = (avg log line size) × (logs per second) × 86400

Storage needed = daily volume × retention days × (1 + replica count) × 1.1 (overhead)

Example:

Parameter	Value
Average log line	500 bytes
Logs per second	5,000
Daily volume	500 × 5,000 × 86,400 = ~200 GB/day
Retention	30 days
Replicas	1
Total storage	200 × 30 × 2 × 1.1 = ~13.2 TB

With hot-warm-cold:

Tier	Days	Storage Type	Cost Estimate
Hot	0-2	SSD (gp3)	High $/GB
Warm	2-14	Standard SSD	Medium $/GB
Cold	14-30	HDD (st1)	Low $/GB
Snapshot	30-90	S3	Lowest $/GB

Quick Reference

Concept	Details
Storage for ES	PersistentVolumeClaims (never emptyDir)
Hot tier disk	SSD (gp3, fast-ssd)
Warm tier disk	Standard SSD or HDD
Cold tier disk	Cheapest HDD (st1, sc1)
ILM	Automates rollover → shrink → delete
SLM	Snapshots to S3/GCS before deletion
Namespace routing	Index per namespace for per-team retention
Legacy tool	Curator (pre-6.6 clusters only)
Planning formula	`daily_vol × days × (1+replicas) × 1.1`

Next: Practice Exercises — hands-on labs to reinforce everything in this chapter.

Share on

Twitter Facebook LinkedIn