Debugging a Managed CoreDNS Regression

Overview

A customer-facing document API on a large, multi-tenant AKS cluster started timing out intermittently. Nobody had deployed anything and no infrastructure had changed. Within a few hours I'd tied the onset to a provider-managed CoreDNS image update rolled out as part of routine add-on maintenance. I also found that a liveness probe in a downstream service was turning short DNS blips into a restart storm.

Why This Was Tricky

• "No change" on our side: no deploys, no config changes, no infrastructure changes in the window.
• Symptoms that looked like the app: 503s, crash-looping pods and slow responses.
• Plausible wrong answers: a single-replica service, CPU-starved burstable nodes and database pressure all fit at first glance.

Architecture

A consumer service in another namespace health-checked the API. Its liveness probe included that dependency check, which turned out to matter more than anyone expected.

Investigation

1. Build the timeline before forming theories

The ReplicaSet history in kube-system showed a new CoreDNS ReplicaSet with a patch-level image bump. The first errors appeared within about two minutes of it rolling out.

kubectl -n kube-system get rs -l k8s-app=kube-dns \
  -o custom-columns=NAME:.metadata.name,CREATED:.metadata.creationTimestamp,IMAGE:.spec.template.spec.containers[0].image

2. Compare against a baseline

The same daily window had hundreds of errors on the incident day and zero on each of the two days before. That pattern points to a change, not load or slow capacity drift.

3. Read CoreDNS's own telemetry

The new pods logged a steady stream of forward health-check failures and upstream i/o timeouts. Most were AAAA and HTTPS record lookups.

4. Disprove the alternatives with data

5. Find the amplifier

The consumer's liveness probe called the API's health endpoint, and failures were never cached. Every DNS blip made kubelet restart the consumer, and every probe sent a live request to one of the API's busiest endpoints.

Recommendations

1. Decouple liveness from dependencies. Liveness answers "is this process alive?". Dependencies belong in readiness.
2. Escalate to the provider. Rolling back proves causation, but the add-on manager reconciles the image back, so the fix has to come from them.
3. Add DNS headroom: NodeLocal DNSCache, ndots: 2 for high-volume external lookups, and a higher autoscaler ladder for CoreDNS.
4. Customise only through coredns-custom. Direct edits to the managed ConfigMap get reverted.

# Illustrative: cut search-domain fan-out for workloads making many external calls
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

Key Outcomes

• Root cause tied to a specific managed change, with onset timing and a zero baseline.
• Avoided two costly non-fixes: scaling the API and resizing the nodes.
• Flagged that the provider's maintenance cadence would bring the issue back, before it recurred.
• Raised several unrelated hygiene findings uncovered along the way.

Lessons Learned

• My first hypothesis was wrong. Writing down what the data rules out is as valuable as the root cause.
• On managed Kubernetes, "we changed nothing" doesn't mean nothing changed. Check add-on history first.
• A liveness probe that checks other services turns their outage into your restart storm.

Conclusion

Evidence first, theory second. The obvious fix would have cost money and changed nothing. The lasting fixes were in probe design and DNS headroom.

Technologies Used