Crash-Loops with Three DNS Root Causes

Overview

Three newly deployed services had never become Ready, and each had racked up around a thousand restarts. Their logs all said the same thing: health check failed. Testing from inside each pod, one network layer at a time, found three separate DNS problems, and a redeploy wouldn't have fixed any of them.

Why It Was Hard

The startup probe killed each pod every few minutes, so containers never stayed alive long enough to debug comfortably.
Names resolved fine from engineers' laptops, so "it works for me" kept sending people in the wrong direction.
The cluster, VNets, corporate DNS and private DNS zones each belonged to a different team.

Architecture

A record only helps a pod if its zone is reachable through this chain

Method: Test Every Layer from Inside

run from a debug pod in the same namespace

getent hosts <fqdn>                                   # DNS  - what does the pod resolve?
nc -zv <ip> 443                                       # TCP  - is the port reachable?
openssl s_client -connect <fqdn>:443 </dev/null       # TLS  - is it the right endpoint?
wget -qO- https://<fqdn>/health                       # HTTP - does the app answer?

Service	What failed	Root cause
Auth-dependent worker	The identity provider's hostname was NXDOMAIN inside the cluster only	The company domain lived on on-prem DNS, and the cluster VNet's resolver didn't serve that zone
Notifications worker	Blob hostname NXDOMAIN, yet TCP 443 to the private IP was open	The record existed, but its zone was linked to a different VNet. A missing link, not a firewall
Event-stream consumer	Name resolved and TCP connected, but the TLS handshake failed	Several duplicate `privatelink.servicebus` zones held conflicting records, and the cluster got a stale IP

Recommendations

Serve or forward the company domain through the cluster's DNS path.
Register private endpoint records in the single zone the corporate resolver treats as authoritative, instead of adding ad-hoc VNet links.
Reduce duplicate privatelink zones to one per service type, and audit them regularly.
Make the pod-level connectivity checks above a standard runbook for every new private endpoint.

Azure Resource Graph

// Find duplicate privatelink zones across subscriptions
resources
| where type =~ 'microsoft.network/privatednszones'
| where name startswith 'privatelink.'
| summarize copies = count(), groups = make_set(resourceGroup) by name
| where copies > 1

Key Outcomes

Three distinct root causes isolated in one read-only session, without touching production.
Showed that the fixes belonged to the network and DNS owners, which ended a cycle of pointless redeploys.
Surfaced duplicate-zone sprawl as an estate-wide risk.

Lessons Learned

Always test DNS from where the workload runs.
"TCP open, TLS fails" almost always means you've reached the wrong endpoint.
Duplicate private DNS zones fail silently and unpredictably. Treat them as technical debt.

Conclusion

One symptom hid three causes. Methodical, layer-by-layer testing separated them quickly and turned three tickets into one platform fix.

Technologies Used

AKSCoreDNSStartup ProbesAzure Functions on KubernetesAzure Private DNSPrivate EndpointsEvent HubsBlob StorageAzure Resource Graph

About this case study: this write-up is drawn from my hands-on platform engineering experience. Organisation names, system names, figures and timelines have been changed or generalised, and all code, configuration and queries are illustrative examples written for this page.

← Back to Projects Get in Touch