Multi-Tenant Environment

📝 Context

Customer is running a multi-tenant Kubernetes cluster where multiple teams or applications share the same cluster. This introduces isolation, resource management, governance, and security considerations beyond single-tenant deployments. Validate both the technical controls and the operational process for requesting exceptions or additional capacity.

%%{init: {'theme': 'neutral', 'themeVariables': {'fontSize': '14px'}}}%%
flowchart TB
    Cluster[Shared Kubernetes cluster] --> TenantA[Tenant namespace]
    Cluster --> TenantB[Other tenant namespace]
    Cluster --> Shared[Shared platform services]

    TenantA --> Workloads[Application workloads]
    TenantA --> Quota[Resource quota and limits]
    TenantA --> RBAC[RBAC and service accounts]
    TenantA --> NetPol[Network policies]
    TenantA --> Admission[Admission policies]

    Shared --> Ingress[Ingress and certificates]
    Shared --> Observability[Logs and metrics]
    Shared --> Registry[Registry and image policy]

🎯 Discovery Questions

• How is multi-tenancy implemented? (Namespaces, virtual clusters, other?)
• What isolation mechanisms are in place? (Network policies, RBAC, resource quotas?)
• Are there shared services or resources we need to be aware of?
• What are the resource quotas and limits for our namespace?
• Are there cluster-level policies that affect all tenants?
• How is network isolation handled between tenants?
• What's the process for requesting additional resources?
• Are there any tenant-specific restrictions or requirements?
• Who owns admission policies, pod security standards, and image policies?
• Are noisy-neighbor, priority class, or preemption controls configured?
• What tenant-level audit, cost allocation, and chargeback requirements exist?

🏗️ Architecture Questions

• Is the tenant boundary a namespace, a set of namespaces, a virtual cluster, or a dedicated node pool?
• Which cluster-scoped resources can the implementation use, and who must create them?
• What ingress, service mesh, DNS, certificate, and secret-management patterns are approved?
• Are tenants isolated by network policies, CNI features, service mesh policy, or firewall rules?
• Are workloads constrained by pod security admission, OPA/Gatekeeper, Kyverno, SCCs, or custom admission webhooks?
• Are dedicated nodes, taints/tolerations, runtime classes, or storage classes required for this tenant?
• How are shared dependencies versioned, upgraded, and communicated to tenants?
• What backup, restore, and disaster recovery boundaries exist for tenant data?
• What telemetry can tenant operators see, and what remains cluster-admin-only?

🎯 Scoping Considerations

• Add 20-30% time buffer for multi-tenant deployments
• Account for namespace and resource quota setup time
• Understand isolation requirements and constraints
• Plan for potential resource contention issues
• Consider network policy configuration needs
• Account for coordination with other tenants if needed
• Understand shared service dependencies
• Include time for security review of RBAC, service accounts, images, and admission policy exceptions
• Confirm whether cluster-scoped components require a separate platform-team workstream
• Build quota increase and policy exception lead times into the schedule

✅ Validation Checklist

• [ ] Target namespace or virtual cluster exists and is labeled according to customer standards
• [ ] RBAC permissions are verified for deployer, runtime service accounts, and support users
• [ ] ResourceQuota and LimitRange values cover expected steady-state and rollout surge needs
• [ ] Network policies allow required ingress/egress and block unintended tenant traffic
• [ ] Pod security, admission, image-signing, and registry policies are tested before deployment
• [ ] Required storage classes, ingress classes, certificate issuers, and secret providers are available
• [ ] Observability access is available for tenant operators without requiring cluster-admin privileges
• [ ] Backup/restore responsibilities and tenant data boundaries are documented
• [ ] Quota exhaustion, pod eviction, and failed admission scenarios are tested or simulated
• [ ] Escalation path exists for cluster-scoped changes and emergency quota/policy changes

🎯 Customer Communication

• Clarify isolation requirements and expectations
• Understand resource quota approval process
• Discuss network policy requirements early
• Set expectations about shared resource usage
• Establish process for requesting quota increases
• Clarify any tenant-specific compliance requirements
• Explain which tasks require tenant-admin access versus cluster-admin access
• Confirm how shared-service maintenance or incidents are communicated to tenants

⚠️ Gotchas

• Resource quotas can cause failures - understand limits upfront
• Network policies can block connectivity - test early
• Shared services may have different SLAs - clarify expectations
• Other tenants' actions can affect you - understand isolation guarantees
• Quota increases may require approval - plan ahead
• Namespace setup may require cluster admin - identify who can help
• Isolation mechanisms vary - understand exactly what's in place

🚨 Common Failure Modes

• Helm installs fail because charts attempt to create cluster-scoped resources the tenant cannot create
• Pods are rejected by admission policies for security context, image source, or missing labels
• Rollouts fail even though steady-state fits because quotas do not allow surge capacity
• Network policies allow pod-to-service traffic but block DNS, metrics, webhook, or registry access
• Shared ingress or certificate controllers require annotations that differ from upstream examples
• Tenants cannot access logs or metrics needed to troubleshoot without platform-team assistance
• Node-level contention, eviction, or priority settings create intermittent failures
• Backup or restore scope is misunderstood, causing tenant data recovery gaps

📊 Required Diagrams / Artifacts

• Tenant boundary diagram showing namespaces, virtual clusters, node pools, and shared services
• RBAC matrix for deployer, runtime service accounts, support users, and cluster admins
• Resource quota and limit worksheet with steady-state, rollout, and failure-recovery calculations
• Network policy matrix with source, destination, port, protocol, and tenant isolation intent
• Cluster-scoped dependency inventory with owner, approval path, and change window
• Admission policy exception register, if any exceptions are required
• Tenant operations runbook covering quota requests, incident escalation, and shared-service changes

🗣️ Customer Expectation-Setting Language

In a multi-tenant cluster, the namespace is not the only boundary that matters. We need to validate RBAC, quotas, network policy, admission controls, shared services, and support access before we treat the environment as ready.

Some deployment steps may require platform-team or cluster-admin action. We will identify those steps early so the implementation timeline does not depend on last-minute exceptions.

Resource quotas need to cover rollout and recovery behavior, not just steady-state usage. We should size for upgrades, restarts, and failure recovery before production cutover.

🔗 Technical Resources

• Multi-Tenant Validation Template
• Environment Validation
• implementation-studio