We ran an outbound platform across Email, SMS, App, and WhatsApp. What we sent at the source did not match what we could account for at the target. Between “queued” and “delivered,” the numbers quietly drifted.
That gap is worse than it looks. When delivery numbers and logged numbers disagree, you cannot tell a real failure from a logging artifact. Every investigation starts with an argument about which layer is lying. You debug the instrument instead of the system.
What the work actually was
Not a rewrite. Making the numbers trustworthy.
- Count each stage on its own so one aggregate cannot hide the break.
- Walk the hops until you find where the count diverges.
- Kill false drops: requests logged as failures that had already been delivered. Those poison every signal.
We got to 100% accounting of message counts from source to target. A number you could defend.
What I still believe
Observability is not a wall of dashboards. It is counts you can trust. A metric you do not believe is worse than no metric. It sends you to the wrong layer with confidence.
Before you optimize a pipeline, make it auditable. If you cannot say where each message went, you are not operating the system. You are hoping.