Cost anomaly alerts have a deservedly bad reputation: pages you at 2am for a deploy that legitimately doubled a workload. When we rebuilt our own alerting, we had one engineer and two weeks. This is what shipped.
The problem with naive thresholds
Cloud spend is dominated by seasonality: daily peaks when batch jobs run, weekly dips on weekends, monthly spikes when reserved capacity renews. A flat threshold over a rolling average learns exactly one lesson — cry wolf.
Seasonal decomposition first, ML later
We decompose hourly cost into trend, day-of-week, and hour-of-day components using classic STL-style averaging over a trailing 4-week window. The residual is what gets thresholded — with a fixed z-score of 4.5 rather than anything learned. Boring, auditable, and it survives on-call reviews.
The 70% false-positive reduction came from the decomposition, not from a fancier model. Fix the baseline before touching the detector.
The rules layer nobody admits they need
- Deploy awareness: alerts raised within 90 minutes of a service deploy route to the deploying team's channel, not on-call.
- Sustained-only paging: a residual must persist for 3 consecutive hours before paging; one-off spikes go to a digest.
- Unit economics guard: normalize by events processed — cost per million events is the number that actually matters.
Where to go next
We're experimenting with a simple Prophet-style model for the trend component, mostly to handle product launches that shift the baseline stepwise. Early results are encouraging; the rules layer is doing the heavy lifting regardless.
Questions or want to compare setups? Write to us — we read everything.