Alert Fatigue Is an Outage Risk: Cutting Noisy Alerts with SLO-Based Alerting - SysRoot

The Problem Nobody Wants to Admit

Ask any on-call engineer at a growing company how they feel about their alerts, and you will hear some version of the same answer: most of them are ignored. The pager fires at 3 a.m. for a CPU spike that self-resolves in ninety seconds. A disk hits 80% and screams every fifteen minutes for a week. A flapping health check pages four people simultaneously for a service that never actually went down.

This is alert fatigue, and it is not a minor annoyance. It is a reliability risk. When 90% of your pages are noise, your team learns to swipe them away without reading. The one page that actually matters — the database running out of connections, the checkout API returning 500s — gets the same reflexive dismissal. The cost of noisy alerting is not measured in interrupted sleep. It is measured in the real incident you missed because it looked like all the others.

This article is about one specific fix: moving from cause-based, threshold-driven alerting to SLO-based alerting built on symptoms and error budgets. It is the single highest-leverage change most small teams can make to their monitoring stack.

Why Threshold Alerts Generate Noise

Most alerting starts the same way. You install a monitoring agent, it ships a default dashboard, and someone wires up alerts on the metrics that are easy to measure: CPU, memory, disk, load average. These are cause metrics — they describe the internal state of a machine, not the experience of a user.

The trouble is that cause metrics correlate poorly with actual problems.

High CPU is not an incident

A service pinned at 95% CPU might be perfectly healthy — it is doing work, requests are fast, users are happy. Another service at 30% CPU might be completely broken because it is deadlocked waiting on a database. If you alert on CPU, you page on the first case and miss the second.

Static thresholds age badly

You set a disk alert at 80% in March. By September the service has grown, the baseline is now 78%, and the alert fires constantly during normal operation. Nobody updates it, so it becomes background noise. Worse, when the disk genuinely heads toward 100%, the alert that should have warned you has already been muted.

Every cause gets its own alert

Threshold-based monitoring encourages an alert per metric per host. A ten-node cluster with twenty metrics each produces two hundred potential alert sources. During a real incident, a single root cause cascades into dozens of simultaneous pages — a phenomenon called an alert storm — burying the signal you need under the symptoms you do not.

The result is predictable: too many alerts, most of them not actionable, and a team that has stopped trusting the pager.

The Core Shift: Alert on Symptoms, Not Causes

The principle behind SLO-based alerting is simple to state and surprisingly hard to adopt: page humans only for things that affect users, and diagnose causes after you are already looking.

A symptom is something a user would notice: requests are failing, requests are slow, the page does not load. A cause is the underlying mechanism: the database connection pool is exhausted, a node ran out of memory, a deploy introduced a bad query.

You still collect cause metrics — you need them to debug. But you do not page on them. They live on dashboards and feed into diagnosis. Pages are reserved for symptoms that breach an objective you have explicitly agreed is important.

This is where Service Level Objectives come in.

SLIs, SLOs, and error budgets in plain terms

An SLI (Service Level Indicator) is a measurement of something users care about, expressed as a ratio of good events to total events. The most common ones are availability and latency:

• Availability SLI: successful requests / total requests
• Latency SLI: requests faster than 300ms / total requests

An SLO (Service Level Objective) is the target you commit to — for example, 99.9% of requests succeed over a rolling 28-day window.

The error budget is what is left over: 100% minus your SLO. A 99.9% target gives you a 0.1% error budget. Over 28 days, that is roughly 40 minutes of total downtime, or a fixed number of failed requests, that you are allowed to spend before you are out of compliance.

The error budget reframes everything. Instead of asking "is CPU high?", you ask "are we burning our error budget faster than we can afford?" That question maps directly to user pain, and it is the basis for alerts that actually matter.

Burn-Rate Alerting: The Practical Mechanism

The naive way to use an SLO for alerting is to page when you have burned, say, 5% of your error budget. This is too slow for fast outages and too jumpy for slow ones. The better approach, popularized by Google's SRE practice, is burn-rate alerting.

Burn rate is how fast you are consuming your error budget relative to the rate that would exhaust it exactly at the end of the window. A burn rate of 1 means you will use the whole budget precisely over 28 days. A burn rate of 14.4 means you are burning budget fourteen times too fast — at that rate you would exhaust a month's budget in about two days.

The trick is to use multiple burn-rate windows with different sensitivities:

1. Fast burn (page now): A high burn rate (e.g. 14.4) sustained over a short window (e.g. 1 hour, confirmed by a 5-minute window). This catches sharp, severe outages — a bad deploy taking down 30% of traffic. It pages immediately.
2. Slow burn (ticket or low-priority page): A lower burn rate (e.g. 3) over a longer window (e.g. 6 hours). This catches the slow leak — a steady 1% error rate that no single threshold would flag but that will quietly drain your monthly budget.

The two windows together give you the best of both: fast detection for big problems, patient detection for slow ones, and far fewer false positives. A transient blip that resolves in ninety seconds never sustains a high burn rate long enough to page, so the 3 a.m. CPU spike simply never wakes anyone.

A Realistic Scenario

Consider a mid-sized SaaS company running a payments API on Kubernetes. Their old setup had alerts on pod restarts, node CPU, memory pressure, and HTTP 5xx count per pod. On a typical week the on-call engineer received 60 to 80 pages, of which maybe two were real.

One afternoon a deploy introduced a slow query. The connection pool started saturating. Under the old system, here is what happened: memory alerts fired on three pods, CPU alerts fired on two nodes, pod-restart alerts fired as the liveness probe killed saturated pods, and the 5xx counter alert fired per pod. Fourteen pages in four minutes, to three different people. By the time anyone untangled which alert was the cause and which were effects, the API had been returning errors for twenty minutes.

With SLO-based alerting, the same incident looks different. The availability SLI for the payments API drops from 99.99% toward 97%. The 5-minute and 1-hour burn-rate windows both cross the fast-burn threshold within two minutes. One page fires: "Payments API burning error budget at 18x — fast burn." The engineer opens the service dashboard, sees the 5xx rate climbing alongside the connection-pool saturation graph (a cause metric, on the dashboard, not paging), correlates it with the deploy timestamp, and rolls back. One page, clear signal, faster resolution.

The cause metrics did not disappear. They were exactly where they needed to be — in front of the engineer who was already looking — instead of competing to wake people up.

What You Still Need Cause Metrics For

SLO-based alerting does not mean throwing away your infrastructure metrics. It means changing their job. Cause metrics belong in two places:

• Dashboards organized per service, so that when a symptom-based page fires you can immediately see the likely cause.
• Capacity and trend alerts that go to a ticket queue, not a pager. "Disk will be full in 4 days at current growth" is a real signal, but it is a Tuesday-morning task, not a 3 a.m. emergency. Route it accordingly.

The discipline is about routing severity correctly: pages for user-facing symptoms, tickets for things that need attention but not immediate action, and dashboards for everything you might need to debug.

A Migration Checklist

Moving an existing noisy stack to SLO-based alerting is incremental. You do not need to rebuild everything at once.

1. Pick your most critical user-facing service first. Do not try to SLO everything on day one. Start where an outage hurts most.
2. Define one or two SLIs per service. Availability and latency cover most cases. Measure them at the point closest to the user — ideally your load balancer or API gateway, not inside individual pods.
3. Set an honest SLO. Do not default to 99.999%. Look at your last few months of real performance and pick a target slightly better than your current reality. An unrealistic SLO produces constant burn-rate pages and recreates the fatigue you are escaping.
4. Implement multi-window burn-rate alerts. Start with a fast-burn page and a slow-burn ticket. Most monitoring platforms with a query language can express these as ratios over time windows.
5. Move all cause-based alerts off the pager. Reclassify CPU, memory, disk, and restart alerts as dashboard panels or low-priority tickets. This is the step that actually reduces the noise.
6. Run both systems in parallel for two weeks. Watch whether the SLO alerts catch the real incidents the old alerts caught. Tune thresholds before you delete anything.
7. Review error budget consumption monthly. If you keep blowing the budget, you have a reliability problem to invest in. If you never touch it, your SLO may be too loose or you are over-investing in reliability nobody needs.

Common Failure Modes

A few mistakes recur often enough to call out:

• SLOs that are too strict. Targeting 99.99% when your real availability is 99.5% means the burn-rate alerts fire constantly. You will be back to fatigue within a week.
• Measuring at the wrong layer. If your SLI is computed from per-pod metrics, a single restarting pod can distort the ratio. Measure aggregate user-facing success at the edge.
• Forgetting the slow-burn window. Teams often implement only the fast-burn page and miss the gradual degradations that quietly drain the budget over days.
• Treating the error budget as a number for dashboards only. Its real value is as a decision tool: when the budget is spent, you slow down risky changes; when it is healthy, you ship faster. Without that policy, the SLO is just decoration.

The Payoff

The goal is not fewer alerts for their own sake. It is a pager you trust. When every page corresponds to real user pain and most of the noise has moved to dashboards and tickets, on-call stops being a punishment and starts being effective. The engineer who gets paged knows it matters, looks immediately, and has the cause metrics waiting where they are needed.

That trust is the whole point. Monitoring that nobody believes is worse than no monitoring at all, because it gives you false confidence while training your team to ignore the one signal that counts.

---

Struggling with Monitoring & Observability?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Monitoring & Observability, talk to the SysRoot team and we'll help you get it under control.