Amazon EKS — FSx for Lustre

1 minute read

FSx for Lustre

Amazon FSx for Lustre is a high-performance parallel file system designed for ML training, HPC, and media processing workloads.

When to Use FSx for Lustre

Workload Storage Choice
General apps EBS (gp3)
Shared files EFS
ML training (large datasets) FSx for Lustre
HPC / simulation FSx for Lustre
Video processing FSx for Lustre

Key Features

Feature Value
Throughput Up to hundreds of GB/s
IOPS Millions
S3 integration Automatic sync with S3 bucket
Access mode ReadWriteMany
Pricing Starting at $0.14/GB/month

Install FSx CSI Driver

# Install via Helm
$ helm repo add aws-fsx-csi-driver https://kubernetes-sigs.github.io/aws-fsx-csi-driver
$ helm install aws-fsx-csi-driver aws-fsx-csi-driver/aws-fsx-csi-driver \
    --namespace kube-system

Use FSx for Lustre

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fsx-lustre
provisioner: fsx.csi.aws.com
parameters:
  subnetId: subnet-0abc123
  securityGroupIds: sg-0def456
  deploymentType: PERSISTENT_2
  perUnitStorageThroughput: "200"    # MB/s per TiB
  dataCompressionType: LZ4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ml-data
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: fsx-lustre
  resources:
    requests:
      storage: 1200Gi    # Minimum 1.2 TiB

S3 Integration

FSx for Lustre can be linked to an S3 bucket for transparent data access:

S3 bucket ←──sync──→ FSx for Lustre ←──mount──→ Pods

Training data stays in S3, FSx provides high-speed access.

Common Mistakes

Mistake Impact
Using FSx for general workloads Over-engineered and expensive
Minimum 1.2 TiB size Can’t create smaller volumes
Wrong subnet/security group Mount fails

What’s Next?

Practice: Storage Exercises.

You may also enjoy