An ops team came to us with a familiar problem: hundreds of alerts a day, almost all noise. Their on-call engineers were drowning in pages, and real incidents were getting buried in the flood.
This is what happens when alerting is built around static thresholds. CPU above 80%? Alert. Response time above 500ms? Alert. The thresholds don’t understand context — they don’t know that 80% CPU is normal during your batch processing window, or that 500ms response times are expected during a deploy.
The situation
The team had a mature monitoring stack. Dashboards, metrics collection, log aggregation — the infrastructure was solid. The problem was the alerting layer on top of it.
Their alert rules had accumulated over years. Every incident led to a new threshold. Every false positive led to a tweak. The result was a system that was simultaneously too noisy (hundreds of false positives) and too slow (real incidents took an average of 12 minutes to detect, because they were buried in the noise).
What we built
We didn’t rip out their monitoring stack. We replaced the alerting layer with something smarter.
Anomaly detection instead of thresholds. Instead of “alert when CPU > 80%,” the system learns what normal looks like for each metric at each time of day and day of week. It alerts when something is genuinely unusual — not just when a number crosses an arbitrary line.
Alert correlation. When five metrics go weird at the same time, that’s probably one incident, not five. The system groups related anomalies and surfaces a single alert with context about all the signals that triggered it.
Feedback integration. When an on-call engineer acknowledges, escalates, or dismisses an alert, that signal feeds back into the system. Over time, it learns which patterns matter and which are just noise.
The architecture
This is where “AI-native” matters. We didn’t build a separate ML system that consumes metrics via a batch export. The anomaly detection runs on the same data pipeline as their existing dashboards. The alert correlation happens in real time, not in a post-processing step. The feedback loop is part of the alert workflow, not a separate data collection effort.
The key design decisions:
- Streaming inference on the existing metrics pipeline, so detection latency is seconds, not minutes
- Model updates run through the same CI/CD as their other deployments, with canary testing
- Alert routing integrates with their existing PagerDuty setup — no new tools to learn
Results
After two weeks of calibration:
- 70% fewer alerts. Most of the reduction was false positives that the threshold-based system couldn’t distinguish from real issues.
- 12 minutes to under 2. Mean time to detect real incidents dropped because engineers weren’t wading through noise to find the signal.
- On-call satisfaction went up. This one’s hard to quantify, but the team reported that on-call shifts went from dreaded to manageable.
The system has been running for several months now. The anomaly detection has continued to improve as it accumulates more data and feedback, and the team has added new signals without needing to manually define thresholds for each one.
The takeaway
The technology here isn’t novel — anomaly detection and alert correlation have been around for years. What made this work was the architecture. Because the AI components were integrated into the existing monitoring pipeline from the start, they could run in real time, learn from feedback, and be maintained by the same team that runs the rest of the stack.
If we’d bolted this on as a separate system, it would have added complexity without reducing it.