Silent Failure Modes in Shared Helm Charts

Overview

Every team deploys to AKS through one shared Helm chart and one reusable pipeline. While onboarding a new service, I found five behaviours where the deploy reported success but the app was misconfigured, unreachable or not updated at all. I fixed the pipeline, documented the chart's real contract and added checks so other teams don't hit the same traps.

Architecture

Shared deploy path. Each failure mode below happens at one of these stages.

The Five Failure Modes

#	Symptom	Root cause	Fix
1	Per-environment config never loads	The chart ignores `args` and `command`, so the image `CMD` always runs	One config file with `${VAR}` placeholders, filled from env vars
2	Endpoints exist, but requests get no response	Named `targetPort`, but the container port has no name, so the EndpointSlice port is unset	Always use a numeric `targetPort`
3	Deploy succeeds with no new pods	The mutable `:main` tag left the pod spec unchanged	Default the tag to the commit SHA
4	Deploy fails on the namespace	The deploy identity only had namespace-scoped rights	Terraform owns namespaces, and the pipeline skips creating them
5	Frontend changes never ship	The build step only rebuilt the backend bundle	Run both build steps, in order

diagnosing #2

# Is the Service actually bound to a port?
kubectl get endpointslice -l kubernetes.io/service-name=<svc> \
  -o jsonpath='{.items[*].ports}'
# Empty, or a port without a number = the targetPort name didn't resolve

values that work with the shared chart (illustrative)

deployment:
  containerPort: 7007
  env:
    - name: APP_BASE_URL
      value: https://app.dev.example.internal
service:
  targetPort: 7007          # a number, never a name
serviceAccount:
  annotations:
    azure.workload.identity/client-id: <client-id>

Guardrails I Added

Immutable, commit-SHA image tags by default, so every deploy is a traceable rollout.
A written contract for the chart: what it templates and what it silently ignores.
A version-bump check that flags when a new chart release changes args handling.
Recommended defaults for golden-path values (see the developer platform) so new services avoid all five.

Lessons Learned

"Helm succeeded" means the API accepted the objects, not that the app works. Verify endpoints and rollouts.
Mutable tags and declarative deploys don't mix.
Shared charts deserve the same contract, tests and changelog as any shared library.

Conclusion

None of these were hard bugs. They were invisible ones. Making them visible, and making the safe option the default, is what platform engineering is for.

Technologies Used

HelmKubernetesEndpointSlicesAzure Workload IdentityKey Vault CSI DriverGitHub ActionsDockerAzure Container Registry

About this case study: this write-up is drawn from my hands-on platform engineering experience. Organisation names, system names, figures and timelines have been changed or generalised, and all code, configuration and queries are illustrative examples written for this page.

← Back to Projects Get in Touch