Unblocking Batch Jobs: An Admission Webhook Bottleneck

Overview

Teams across the cluster reported the same thing: batch Jobs (crawlers, renderers, scheduled tasks) were taking several minutes to even start, when they normally started in seconds. Working with cloud provider support, we traced it to a mutating admission webhook running as a single, rate-limited replica. I delivered the fix through every environment and found the extra step it needed to actually take effect.

The Symptoms

• Job → pod creation lag of several minutes, and completion lag of a few more, in every namespace.
• Very high churn: KEDA ScaledJobs launching thousands of short Jobs per hour, many lasting under 90 seconds.
• No control-plane diagnostic logs enabled, and the cluster's health status showed healthy.

How Admission Works

Every pod create passes through mutating admission webhooks before the scheduler ever sees it. If one webhook is slow, every controller that creates pods queues behind it, and the Job controller has only a handful of workers by default.

Investigation

1. Measure precisely. I computed Job → pod and Succeeded → Complete deltas per namespace from Kubernetes telemetry. The lag was the same for every team, which points at the platform rather than any one workload.
2. Separate the noise. One node pool was also at its autoscaler maximum. That was real, but it couldn't explain delays for pods that hadn't been created yet.
3. Escalate with evidence. A support case with the timeline and churn numbers led the provider to the injector. Its API client was rate-limited to a few requests per second, so each pod create took around two seconds of admission time.
4. Correct the timeline. The delay had also appeared on earlier days. Low weekend load had hidden it.

The Fix

1. Scaled the sidecar injector from one replica to two through a reviewed change to the cluster core-services repo.
2. Verified in dev and test, including that traffic was split across both replicas.
3. Shipped to production under an emergency change.
4. The extra step: after the production rollout, the delay only halved. The API servers kept their long-lived connections pinned to the original pod. A rolling restart of the injector forced them to reconnect and spread the load.

# After scaling ANY admission webhook, make the API servers rebalance
kubectl -n <injector-namespace> rollout restart deploy/<sidecar-injector>
kubectl -n <injector-namespace> rollout status  deploy/<sidecar-injector>

Key Outcomes

• Delivered through dev, test and production, with verification at each stage.
• Captured the restart-after-scale lesson as platform guidance.

Lessons Learned

• A symptom that's identical in every namespace means a shared component. Check admission webhooks early.
• Scaling a webhook isn't enough. Restart it so the API servers spread their connections.
• Turn on control-plane diagnostics before you need them.

Conclusion

One replica of one webhook was quietly throttling an entire cluster. Measuring precisely, escalating with evidence and checking the result in production turned a vague "everything is slow" into a seconds-level fix.

Technologies Used