If your engineers are woken at 3am by an alert that turns out to need nothing, the problem is usually what the alert is watching. Most alert sets grow one incident at a time: something broke, someone added a line for CPU, memory or error count, and nobody removed it later. Over time people learn to glance and go back to sleep, and the alert that matters gets the same treatment as the rest.
The steps below are standard practice in site reliability engineering (SRE: treating the job of keeping production running as an engineering problem), in the order we’d apply them.
Alert on what users feel, not on what the server is doing
Our default for a small team is to alert on symptoms rather than causes: slow responses rather than CPU at 90%. A server running hot at 90% CPU may be serving every customer perfectly well, and a server at 30% may be returning errors to half of them. Symptoms are what your customers notice, so an alert on a symptom is far more likely to need a person.
For the first 90 days, point your monitoring at your most important service and make sure it answers four plain questions: is it down, is it slow, is it broken (error rates and status codes), and is it driving users away?
Page on how fast you are using your error budget
To know when a symptom is bad enough to wake someone, you need a reliability target. In SRE terms this is an SLO (service level objective: a target such as 99.9% of checkouts succeeding over 30 days). The gap between that target and 100% is the error budget (the amount of failure the target allows), and 99.9% over 30 days allows 43.2 minutes of downtime. The burn rate is how fast you are using that budget; a burn rate of 1 means it lasts exactly the 30 days.
Our default is to turn each SLO into exactly two alerts:
| Alert | Fires when | What happens |
|---|---|---|
| Fast burn | 2% of a 30-day error budget is used within one hour | Someone is paged |
| Slow burn | 10% of a 30-day error budget is used within three days | A ticket is opened for working hours |
We also add shorter checking windows so that an alert clears once the service recovers, rather than staying red for an hour after the problem has gone. With this split, the pager fires only for problems using the month’s allowance quickly; slower leaks still get looked at, just not at 3am.
The business, not engineering alone, should decide what the target is. We recommend against simply copying today’s performance into it, because that records what you happen to have rather than what your customers need. Not every service needs an SLO, and not every SLO needs an alert.
Why fixed thresholds fail on quiet services
A fixed threshold, such as alerting above 0.05% errors on every service, behaves badly at both ends. On a low-traffic service a handful of failures crosses the line and wakes someone, while on a high-traffic service the same line is too loose to catch real trouble. Burn-rate alerts have a version of the same problem: on a service handling only a few requests an hour, two unlucky failures can look like a very high burn rate.
Our fix is a statistical check called the Wilson score interval. In plain English, it asks: given how few requests we have seen, how sure can we be that the error rate is really this high, and not just bad luck? With few requests, the check allows a wide margin, so a few random errors do not page anyone. With lots of traffic, the margin narrows and real problems are caught quickly. As a worked example, confirming with 99% confidence that a 99% SLO is burning at 14.4 times the normal rate (the speed that uses 2% of a 30-day budget in an hour) needs 24 failures out of 100 requests, 42 out of 200, or 173 out of 1,000. We apply this check to services under about 100 requests an hour.
The few cause-based alerts worth keeping everywhere
Symptom alerts do not catch everything. We keep a few cause-based alerts on every service, because by the time users feel these problems it is late, or because your own server metrics will not show them at all:
- quota and resource exhaustion, such as hitting a cloud account limit or running out of disk
- expired or expiring SSL certificates
- DNS and load balancer failures, which happen before a request ever reaches your servers
We set these up as “horizontal” alerts (one definition applied to every service) rather than service by service, because configuring them one at a time leaves gaps wherever a service was missed.
Fix any alert that is rarely right
Noisy alerts are a form of toil (repetitive manual work that grows with the system), and our rule of thumb is that an alert that is right only about one time in 20 should be fixed, so that whoever receives it can trust that action is needed. Watch for missing pages as well as noisy ones, since an alert set can be quiet because it is blind.
We always add a heartbeat (sometimes called a dead man’s switch, an alert that fires when expected signals stop arriving). Monitoring should not fail along with the system it watches, and without a heartbeat a broken alert pipeline looks exactly like a calm month.
Three numbers to track
We judge whether alerting is healthy with three measures:
- Share of unactionable alerts. Pages where nobody needed to do anything. A high share means noisy monitoring.
- Share of incidents first reported by users. Problems that customers or other teams spotted before an alert did. A high share means your alerts are missing what users feel.
- Alerts per incident. How many pages one problem produces. A high number means one fault sets off a storm of related alerts.
Track them monthly. Better detection can make it look as though you have more incidents at first, because problems that were always there start showing up.
How we apply this
Alert clean-up is part of the reliability work in every Tracefox contract: we page on how fast a service is using its agreed allowance for failure, and quiet services get a statistical check so a few random errors don’t wake anyone. Your product and business owners decide the targets, and we rebuild alerts around them by day 90. Your monthly report shows how many pages nobody needed to act on and how many incidents your users spotted before an alert did. See how our reliability work runs.