KCSA — Multi-Tenancy & Isolation

8 minute read

What Is Multi-Tenancy in Kubernetes?

Multi-tenancy is the practice of running workloads from multiple teams, customers, or environments on a shared Kubernetes cluster. The core challenge is ensuring that one tenant cannot access, interfere with, or exhaust the resources of another tenant.

Kubernetes was not originally designed for multi-tenancy. Every isolation mechanism must be explicitly configured — there is no “enable multi-tenancy” flag.

Multi-Tenancy Models

There are three primary models for multi-tenancy in Kubernetes. Each offers different levels of isolation, complexity, and cost.

Model Isolation Complexity Cost Use Case
Namespace per tenant Soft Low Low Internal teams sharing a cluster
Virtual clusters (vCluster) Medium Medium Medium Platform teams, SaaS providers
Cluster per tenant Hard High High Strict compliance, regulated industries

Namespace per Tenant (Soft Tenancy)

Each tenant gets one or more namespaces in a shared cluster. Isolation is enforced using Kubernetes-native mechanisms (RBAC, NetworkPolicy, ResourceQuota).

┌──────────────────────────────────────────┐
│              Shared Cluster               │
│  ┌──────────┐  ┌──────────┐  ┌────────┐ │
│  │ Tenant A │  │ Tenant B │  │Tenant C│ │
│  │ (ns: a)  │  │ (ns: b)  │  │(ns: c) │ │
│  └──────────┘  └──────────┘  └────────┘ │
│                                          │
│  Shared: nodes, CRDs, cluster roles,    │
│          persistent volumes, API server  │
└──────────────────────────────────────────┘

Pros: Low cost, simple to manage, fast onboarding Cons: Noisy neighbor risk, shared kernel, cluster-scoped resources are shared

Virtual Clusters (Medium Tenancy)

Each tenant gets a virtual cluster — a lightweight, isolated Kubernetes control plane running inside the host cluster. The virtual cluster has its own API server, controller manager, and etcd, but shares the underlying worker nodes.

┌──────────────────────────────────────────┐
│              Host Cluster                 │
│  ┌──────────────────┐  ┌──────────────┐ │
│  │ vCluster A       │  │ vCluster B   │ │
│  │ ┌──────────────┐ │  │ ┌──────────┐ │ │
│  │ │ API Server   │ │  │ │API Server│ │ │
│  │ │ etcd         │ │  │ │etcd      │ │ │
│  │ │ Controller   │ │  │ │Controller│ │ │
│  │ └──────────────┘ │  │ └──────────┘ │ │
│  │ Tenant A pods    │  │ Tenant B pods│ │
│  └──────────────────┘  └──────────────┘ │
│                                          │
│  Shared: worker nodes, container runtime │
└──────────────────────────────────────────┘

Pros: Each tenant gets admin-like experience, custom CRDs per tenant, stronger isolation than namespaces Cons: More resource overhead, shared nodes still mean shared kernel

Cluster per Tenant (Hard Tenancy)

Each tenant gets a completely separate Kubernetes cluster. There is no shared infrastructure beyond the cloud account level.

Pros: Strongest isolation, full control per tenant, no noisy neighbor Cons: Highest cost, operational complexity, cluster sprawl

For the KCSA exam, know the trade-offs between these three models. The exam may describe a scenario and ask which model is most appropriate.

Namespace-Level Isolation Mechanisms

When using the namespace-per-tenant model, you must configure multiple isolation mechanisms to achieve meaningful tenancy boundaries.

1. RBAC per Namespace

Bind Roles (not ClusterRoles) to tenants within their own namespace. Ensure tenants cannot access other namespaces.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tenant-a-developer
  namespace: tenant-a
rules:
- apiGroups: ["", "apps", "batch"]
  resources: ["pods", "deployments", "jobs", "configmaps", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-a-developer-binding
  namespace: tenant-a
subjects:
- kind: Group
  name: team-a
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: tenant-a-developer
  apiGroup: rbac.authorization.k8s.io

Never use ClusterRoleBinding for tenant RBAC — it grants access across all namespaces.

2. NetworkPolicy per Namespace

By default, all pods can communicate with all other pods across namespaces. A default-deny NetworkPolicy isolates tenant traffic.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: tenant-a
spec:
  podSelector: {}       # Applies to all pods in the namespace
  policyTypes:
  - Ingress
  - Egress

After applying default-deny, create explicit allow rules for necessary traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: tenant-a
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector: {}   # Allow traffic from pods in the same namespace
  policyTypes:
  - Ingress

3. ResourceQuota per Namespace

Prevent one tenant from consuming all cluster resources.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "20"
    services: "10"
    persistentvolumeclaims: "5"

4. LimitRange per Namespace

Set default resource requests and limits for containers that don’t specify them.

apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-a-limits
  namespace: tenant-a
spec:
  limits:
  - default:
      cpu: "500m"
      memory: 256Mi
    defaultRequest:
      cpu: "100m"
      memory: 128Mi
    type: Container

5. Pod Security Admission (PSA) Labels per Namespace

Enforce Pod Security Standards at the namespace level.

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

The restricted PSA profile enforces: no privileged containers, no host namespaces, must run as non-root, read-only root filesystem, and more.

Namespace Isolation Summary

Mechanism What It Controls Without It
RBAC API access to resources Tenants can read/modify each other’s resources
NetworkPolicy Pod-to-pod network traffic All pods can communicate across namespaces
ResourceQuota Total resource consumption One tenant can starve others of CPU/memory
LimitRange Per-container defaults Pods without limits can consume unbounded resources
PSA Labels Pod security posture Tenants can run privileged containers

All five mechanisms are required for meaningful namespace-level isolation. Missing any one creates a gap.

Hierarchical Namespaces (HNC)

Hierarchical Namespace Controller (HNC) allows you to organize namespaces in a parent-child hierarchy. Policies and objects propagate from parent to child namespaces automatically.

Why HNC Matters for Multi-Tenancy

Feature Description
Policy inheritance RBAC, NetworkPolicies, ResourceQuotas propagate from parent to child
Self-service namespaces Tenants can create sub-namespaces without cluster-admin access
Organizational mapping Namespace hierarchy maps to team/project structure
Quota delegation Parent namespace quota is subdivided among children

Example Hierarchy

org-platform (parent)
├── team-alpha (child)
│   ├── alpha-dev (sub-namespace)
│   └── alpha-staging (sub-namespace)
└── team-beta (child)
    ├── beta-dev (sub-namespace)
    └── beta-prod (sub-namespace)

Policies set on org-platform are automatically propagated to all child namespaces, ensuring consistent security posture.

HNC is a Kubernetes SIG project (not built-in). It must be installed separately as a controller.

Cluster-Level Resources: The Multi-Tenancy Challenge

Some Kubernetes resources are cluster-scoped and shared by all tenants in namespace-per-tenant models:

Resource Scope Multi-Tenancy Concern
Nodes Cluster Pods from different tenants share the same node (and kernel)
PersistentVolumes Cluster PVs are not namespaced — access control must be managed carefully
ClusterRoles / ClusterRoleBindings Cluster Grant cluster-wide permissions — dangerous if assigned to tenants
CustomResourceDefinitions (CRDs) Cluster All tenants share the same CRD schema — version conflicts possible
IngressClass Cluster Shared ingress controllers may allow cross-tenant routing
StorageClass Cluster All tenants use the same storage classes
Namespaces Cluster Tenants should not be able to create or delete namespaces
PriorityClasses Cluster A tenant with a high-priority class can evict other tenants’ pods

Cluster-scoped resources are the primary reason namespace isolation is considered “soft” tenancy. True isolation requires either virtual clusters or separate physical clusters.

Tenant Isolation Checklist

Use this checklist to verify that your multi-tenant cluster is properly isolated:

# Check Mechanism
1 Tenants can only access their own namespace RBAC (Role + RoleBinding per namespace)
2 Tenants cannot communicate across namespaces by default NetworkPolicy (default-deny per namespace)
3 Tenants have resource limits ResourceQuota + LimitRange per namespace
4 Pods run with restricted security posture PSA labels (restricted or baseline)
5 Tenants cannot create cluster-scoped resources RBAC (no ClusterRoleBinding for tenants)
6 Sensitive workloads are isolated on dedicated nodes Node taints + tolerations + node affinity
7 Secrets are encrypted at rest EncryptionConfiguration on the API server
8 Audit logging captures tenant actions Audit policy with per-namespace filtering
9 Admission policies prevent policy violations OPA/Gatekeeper or Kyverno policies
10 DNS isolation prevents cross-namespace service discovery DNS policies or service mesh segmentation

Virtual Clusters (vCluster)

vCluster (by Loft Labs) creates lightweight virtual Kubernetes clusters inside a host cluster. Each virtual cluster runs its own control plane (API server + controller manager + etcd or SQLite) as pods within a host namespace.

How vCluster Works

  1. A virtual cluster is created inside a host namespace
  2. The vCluster API server, controller, and data store run as pods in the host cluster
  3. Tenants interact with their virtual cluster API server — they see an isolated cluster
  4. Workloads scheduled in the virtual cluster are synced to the host cluster as real pods
  5. The host cluster manages the actual scheduling, networking, and storage

When to Use vCluster

Scenario Why vCluster Helps
Multi-tenant SaaS platform Each customer gets their own “cluster” without provisioning real infrastructure
CI/CD environments Spin up ephemeral clusters for testing, tear them down after
Developer self-service Developers get their own cluster without waiting for platform team
CRD conflicts Each virtual cluster can have its own CRD versions without affecting others
RBAC complexity Tenants can be cluster-admin inside their vCluster without affecting the host

vCluster vs Namespace Isolation

Feature Namespace Isolation vCluster
Tenant experience Limited to namespace Full cluster (own API server)
CRD isolation Shared Isolated per virtual cluster
RBAC isolation Namespace-scoped roles Full RBAC including ClusterRoles
Resource overhead None Some (vCluster control plane pods)
Cluster-scoped resources Shared Isolated (virtual)
Underlying node sharing Yes Yes (still shares host nodes)

vCluster does not provide kernel-level isolation — pods still run on shared host nodes. For kernel-level isolation, combine vCluster with sandboxed runtimes (gVisor, Kata).

Key Takeaways for the Exam

Concept What to Remember
Soft tenancy Namespace per tenant — relies on RBAC, NetworkPolicy, ResourceQuota, PSA
Medium tenancy Virtual clusters (vCluster) — isolated control plane, shared worker nodes
Hard tenancy Cluster per tenant — strongest isolation, highest cost
Namespace isolation Requires RBAC + NetworkPolicy + ResourceQuota + LimitRange + PSA (all five)
Cluster-scoped resources Nodes, PVs, CRDs, ClusterRoles — shared in namespace-per-tenant model
HNC Hierarchical namespaces — policy inheritance, self-service sub-namespaces
vCluster Virtual cluster inside a host namespace — own API server, own RBAC, own CRDs
  • Namespaces alone do not provide security isolation — you must configure RBAC, NetworkPolicy, ResourceQuota, LimitRange, and PSA
  • The exam may describe a scenario and ask which tenancy model is appropriate — look for clues about compliance requirements, cost sensitivity, and isolation needs
  • Cluster-scoped resources are the Achilles’ heel of namespace-per-tenant isolation
  • vCluster is a middle ground — better isolation than namespaces, lower cost than separate clusters

Next: Platform Security — Practice Questions →

Updated:

You may also enjoy