Designing for Node Failure on AKS

Overview

Users of a SaaS product couldn't sign in for roughly half an hour, and two dependent tools failed with it. The service eventually recovered by itself, which made it tempting to move on. I rebuilt the timeline instead, and found three ordinary conditions that together turned one node failure into an outage.

Timeline (relative)

T +	Event
0 min	Last successful sign-in transaction
~1 min	Node memory hits about 94% of allocatable, CPU spikes, the node stops reporting and goes NotReady
1–6 min	Replacement auth pod can't schedule: the node pool is at its autoscaler maximum and in scale-up backoff
~6 min	Platform auto-repair reboots the node
~32 min	Node Ready, auth pod running again
~33 min	Sign-ins succeed

Root Cause

Three ordinary conditions, combined

Single replica: losing one node meant losing authentication completely.
Memory saturation: the node was running hot. Requests were probably set below real usage, which let the scheduler overcommit it.
No room to heal: the autoscaler maximum left nowhere for the evicted pod to go until the same node came back.

Recommended Baseline for Critical Services

Control	Standard	Why
Replicas	At least 2, or 3 for auth and gateways	Survives losing a node
PodDisruptionBudget	`minAvailable: 1`	Survives drains and upgrades
Topology spread	Across nodes and zones	Replicas never share a failure domain
Requests	Set from observed p95 usage	Prevents overcommit
Autoscaler	Maximum with headroom, plus a backoff alert	Lets the cluster reschedule
Node alerts	Working set over 85% of allocatable	Warns before NotReady

tier-1 baseline (illustrative)

apiVersion: apps/v1
kind: Deployment
metadata: { name: auth-service }
spec:
  replicas: 3
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector: { matchLabels: { app: auth-service } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: auth-service }
spec:
  minAvailable: 1
  selector: { matchLabels: { app: auth-service } }

Key Outcomes

A minute-by-minute RCA built from node and APM telemetry.
Showed the outage was a resilience gap, not a code defect.
A tier-1 high-availability baseline proposed for every critical service on the shared cluster.

Lessons Learned

Auto-repair works, but a reboot cycle takes tens of minutes. Self-healing isn't high availability.
Autoscaler limits are a resilience setting, not just a cost setting.
More replicas aren't always the answer. In another incident, scaling would have changed nothing. Diagnose first, then apply the standard.

Conclusion

The node failure was ordinary. The configuration around it is what made it an outage. A clear baseline turns the next node failure into a non-event.

Technologies Used

AKSCluster AutoscalerNode Auto-RepairPodDisruptionBudgetsTopology SpreadRequests & LimitsKubernetes Monitoring

About this case study: this write-up is drawn from my hands-on platform engineering experience. Organisation names, system names, figures and timelines have been changed or generalised, and all code, configuration and queries are illustrative examples written for this page.

← Back to Projects Get in Touch