Amazon EKS — Logging with Fluent Bit

Fluent Bit is a lightweight log processor and forwarder. On EKS, it runs as a DaemonSet on every node, tailing container logs and shipping them to destinations like CloudWatch Logs, S3, OpenSearch, or Kinesis.

---

AWS for Fluent Bit Image

AWS maintains an optimized Fluent Bit image:

public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

---

Installation via Helm

helm repo add eks https://aws.github.io/eks-charts

helm install aws-for-fluent-bit eks/aws-for-fluent-bit \
  --namespace kube-system \
  --set cloudWatchLogs.enabled=true \
  --set cloudWatchLogs.region=us-east-1 \
  --set cloudWatchLogs.logGroupName="/eks/my-cluster/containers" \
  --set cloudWatchLogs.logStreamPrefix="fluent-bit-" \
  --set cloudWatchLogs.autoCreateGroup=true \
  --set serviceAccount.create=true \
  --set serviceAccount.name=fluent-bit-sa \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789:role/FluentBitRole

IRSA Setup

eksctl create iamserviceaccount \
  --cluster my-cluster \
  --namespace kube-system \
  --name fluent-bit-sa \
  --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess \
  --approve

---

Fluent Bit Configuration

Fluent Bit uses a pipeline: Input → Parser → Filter → Output.

Send to CloudWatch Logs

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/fluent-bit/state/flb_container.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kube.var.log.containers.
    Merge_Log           On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off

[OUTPUT]
    Name                cloudwatch_logs
    Match               kube.*
    region              us-east-1
    log_group_name      /eks/my-cluster/containers
    log_stream_prefix   node-
    auto_create_group   true

Multi-Destination (CloudWatch + S3)

[OUTPUT]
    Name                cloudwatch_logs
    Match               kube.*
    region              us-east-1
    log_group_name      /eks/my-cluster/containers
    log_stream_prefix   node-
    auto_create_group   true

[OUTPUT]
    Name                s3
    Match               kube.*
    region              us-east-1
    bucket              my-eks-logs-bucket
    total_file_size     50M
    upload_timeout      10m
    s3_key_format       /logs/$TAG/%Y/%m/%d/%H_%M_%S.gz
    compression         gzip

Send to OpenSearch

[OUTPUT]
    Name                opensearch
    Match               kube.*
    Host                search-my-domain.us-east-1.es.amazonaws.com
    Port                443
    TLS                 On
    AWS_Auth            On
    AWS_Region          us-east-1
    Index               eks-logs
    Type                _doc
    Suppress_Type_Name  On

---

Log Filtering

Only forward production namespace logs:

[FILTER]
    Name    grep
    Match   kube.*
    Regex   $kubernetes['namespace_name'] production

Exclude health check logs:

[FILTER]
    Name    grep
    Match   kube.*
    Exclude log /health|/ready|/live

---

Fargate Logging

Fargate runs a built-in Fluent Bit sidecar. Configure via aws-observability namespace:

kubectl create namespace aws-observability

# fargate-logging.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match *
        region us-east-1
        log_group_name /eks/my-cluster/fargate
        log_stream_prefix fargate-
        auto_create_group true
  filters.conf: |
    [FILTER]
        Name parser
        Match *
        Key_Name log
        Parser docker
        Reserve_Data On

The Fargate pod execution role needs CloudWatch Logs permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/eks/*"
    }
  ]
}

---

Destinations Comparison

Destination	Use Case	Cost	Retention
CloudWatch Logs	Default, ad-hoc queries	$0.50/GB ingested	Configurable (1 day–10 years)
S3	Long-term archive, compliance	$0.023/GB stored	Lifecycle policies
OpenSearch	Full-text search, dashboards	Instance + storage costs	Index lifecycle
Kinesis Data Firehose	Real-time streaming	$0.029/GB	Pass-through

---

Best Practices

1. Use IRSA for Fluent Bit — never rely on node-level IAM for log shipping
2. Set Mem_Buf_Limit — prevents Fluent Bit from consuming all node memory
3. Use log retention policies — set CloudWatch log group retention to avoid unbounded costs
4. Filter noisy logs — exclude health checks, liveness probes, and debug-level logs
5. Compress logs for S3 — use gzip compression to reduce storage costs
6. Tag logs with K8s metadata — namespace, pod name, container name for easy filtering

---