Taking a proof-of-concept portal to production on shared AKS clusters with workload identity, private endpoints and SSO
The developer portal began as a single-replica proof of concept with guest login and nowhere durable to keep its state. I rebuilt how it runs so it could serve the whole engineering organisation. It now runs on shared multi-tenant AKS clusters with no stored credentials, private-only data paths, group-based SSO and its own isolated node pool.
| Area | Proof of concept | Production |
|---|---|---|
| Authentication | Guest login, so everyone was signed in automatically | Entra ID SSO gated on security groups |
| State | In-memory or local | Managed PostgreSQL, zone-redundant in production |
| Docs storage | None | Blob Storage behind a private endpoint |
| Secrets | Kubernetes secrets created by hand | Key Vault via the CSI driver and workload identity |
| Scheduling | Anywhere on the cluster | Dedicated tainted node pool |
| Replicas | 1 | 3 (after fixing a connection ceiling) |
# values.yaml (illustrative)
nodeSelector:
workload: developer-portal
tolerations:
- key: dedicated
operator: Equal
value: developer-portal
effect: NoSchedule
serviceAccount:
annotations:
azure.workload.identity/client-id: <managed-identity-client-id>The groups claim only lists direct memberships in the configuration we used, so people who belonged through a nested group were locked out. I added a server-side fallback that calls Microsoft Graph checkMemberGroups to resolve transitive membership authoritatively. The scaffolder is gated on a separate approvers group.
The default refresh-token cookie lasted more than a year. I capped sessions at a working day.
| Symptom | Root cause | Fix |
|---|---|---|
Secret mount fails with no such host | The private endpoint's DNS record wasn't in the zone the corporate resolver treats as authoritative | Register the record in the authoritative zone through an aliased Terraform provider |
Service and endpoints exist, but curl returns 000 | Named targetPort, but the chart doesn't name the container port | Use a numeric targetPort |
| Per-environment config ignored | The chart silently drops args, so the image CMD always runs | One config file with ${VAR} placeholders, filled from env vars |
remaining connection slots are reserved | About 30 DB connections per pod against a low default max_connections | Bigger SKU and explicit max_connections per environment |
| Deploy succeeds, nothing changes | The mutable :main tag left the pod spec unchanged | Tag images with the commit SHA |
| Deploy fails patching the namespace | The deploy identity only had namespace-scoped admin | Terraform owns namespaces, and the deploy skips creating them |
Most of the effort was at the edges: DNS, chart behaviour, connection limits and identity claims. Writing each one down turned a hard deployment into a repeatable pattern for the next team.
About this case study: this write-up is drawn from my hands-on platform engineering experience. Organisation names, system names, figures and timelines have been changed or generalised, and all code, configuration and queries are illustrative examples written for this page.