Inside-Out Ingress Troubleshooting

Overview

A production web portal showed some users a generic request blocked by policy page instead of the app. Firewall, WAF and DNS changes were all being suggested. Before touching anything, I tested every layer from the inside out. Ingress, the app, the API gateway and identity were all healthy. Something in the client's network path was answering before the cluster ever saw the request.

Architecture

Each layer was tested. The block came from the client-side agent.

Layer-by-Layer Results

Layer	Test	Result
Ingress	`curl -I http://<app>` via the ingress IP	`308` → HTTPS `200`, correct page, no block text
App config	Fetched the SPA's real runtime config file	Only expected hosts, and all of them resolve
API gateway	Called every route without a token	Nearly all returned `401` quickly (healthy). One returned `404`
Identity	OIDC discovery document	`200` in milliseconds
Client path	Compared protocols	The block only happened on plain HTTP, where NGINX always redirects. Something upstream answered first

test from inside the cluster's trust boundary

# Bypass DNS and any client-side interceptor entirely
kubectl -n <ns> port-forward svc/<web-app> 8080:80
curl -s localhost:8080/<runtime-config-path> | jq '.endpoints'

Findings

The platform was healthy at every layer.
The block was client-side. A web security agent on the user's device intercepted plain-HTTP requests.
The NXDOMAIN hostnames were red herrings. The deployed app never used those legacy aliases, but a stale CORS allow-list still listed them.
One real bug: one API route returned 404 even though its backend pods were healthy. It was a gateway routing fault.
Config hygiene: the production SPA config still had a development flag set.

Key Outcomes

Avoided unnecessary production changes to the cluster, firewall and DNS.
Handed the security team a precise, reproducible finding.
Surfaced one genuine routing bug and two hygiene items that had been hidden by the noise.

Lessons Learned

Test from the inside out, and change nothing until a layer actually fails.
For single-page apps, find the real runtime config path. Anything else returns index.html and misleads you.
A 401 from an API gateway is a healthy signal. Know what healthy looks like for each layer.

Conclusion

Sometimes the most valuable result is "it isn't the platform", backed by evidence from every layer, so the right team can fix the right thing.

Technologies Used

AKSNGINX Ingress ControllerKubernetes Serviceskubectl port-forwardOpenID ConnectAPI Gatewaycurljq

About this case study: this write-up is drawn from my hands-on platform engineering experience. Organisation names, system names, figures and timelines have been changed or generalised, and all code, configuration and queries are illustrative examples written for this page.

← Back to Projects Get in Touch