Observability: Noise, Gaps and Blind Spots

Overview

Observability only helps if the signal is right. These three cases each found a different way monitoring was misleading people: too much noise, data in the wrong place, and an average hiding the thing that was on fire.

Case 1: A Health Check Flooding the Logs

Symptom. A production service's most frequent transaction was a 404. It had been returning 404s, and never a success, for weeks, through many redeploys.

1. Traced the caller: a function's health endpoint fetched an API documentation file from that service on every call, with no caching.
2. Predicted the request rate from probe intervals × replicas. The measured rate matched exactly.
3. Proved the route didn't exist in production: the docs route was switched off by the same flag as Swagger.
4. Found the path hard-coded in a shared client library, so no configuration change could move it.

Impact. It was the single largest source of volume in that log partition, and its sub-millisecond responses skewed the service's latency percentiles. Fix: point the check at the service's real liveness endpoint.

Side finding: a tagged request proved a non-production service was calling a production backend because of a copied configuration value.

Case 2: Millions of Dev Logs in the Wrong Place

Symptom: "Please enable logging for dev." Reality: logging already worked, at millions of records a day from dozens of services. Every environment had its own log partition except dev, so dev logs landed in the default partition and never appeared in the picker.

• Created the missing partition to match its siblings, and verified records were arriving.
• Found that partitions weren't managed as code anywhere. I checked every branch of every related repo, because code search only indexes default branches.
• Noted that partition rules aren't retroactive, so historical queries still need the default partition.

Case 3: "Half Full" on a Full Disk

Symptom: a database VM's disk alert showed about 50% used. Reality: the metric averaged across all mounts, including a large temporary disk. The data volume itself was effectively 100% full and still growing every day.

SELECT max(diskUsedPercent) FROM StorageSample
WHERE hostname LIKE '%db%' FACET mountPoint

Key Outcomes

• The biggest single source of log noise identified, with a code-level fix.
• A non-production-to-production dependency found and proven.
• Millions of daily dev records made findable, and a gap in observability-as-code documented.
• A misleading disk alert replaced with a per-mount check.

Lessons Learned

• Health checks generate traffic. Multiply probe rate × replicas × dependencies before you ship one.
• Always confirm where logs land, not just that they're emitted.
• Aggregates hide outliers. Alert on the specific resource that matters.

Conclusion

Good observability isn't more data. It's knowing which numbers to distrust, and checking them.

Technologies Used