Monitoring checks for the problems someone predicted: CPU above 90%, error rate above 1%, a server that stopped answering. You need it, and it’s where every reliable setup starts. But incidents tend to come from the things nobody predicted, and then monitoring can tell you that something is wrong without telling you why, or who it’s happening to.
Observability is the ability to ask your systems new questions in the middle of an incident, without already knowing them inside out. Is it one customer or all of them? One release or every release? Which service did the slow requests spend their time in? This guide covers where monitoring falls short and the order we’d add to it.
Monitoring and observability, in plain terms
Observability isn’t a product you buy. It comes from collecting four kinds of data and being able to connect them:
| Data | What it is | What it answers |
|---|---|---|
| Metrics | Numbers over time, such as requests per second or error rate | Is something wrong right now? Cheap to keep and fast to alert on, but they can’t tell you which customer or which request. |
| Logs | A record of individual events, best written as structured fields (release, customer, endpoint) rather than free text | What exactly happened to these requests, and what do the failing ones have in common? |
| Traces | The path one request takes across your services, with the time spent at each step | Where did the slow or failed request actually go wrong? |
| Profiles | A continuous record of where your code spends CPU and memory | Which part of the code is using the resources? Useful later; most teams start with the first three. |
The point of collecting all four is to reduce how much an engineer needs to know in advance to diagnose a problem. That matters most at 3am, when the person on call didn’t write the code that broke.
Five ways monitoring misses real problems
Your servers report success while users fail
A search that returns no results is still a successful response to the server. A checkout button that won’t submit never sends a request at all. Google has published an example of exactly the first case: its Maps servers returned technically successful responses with zero search results, and the request-level monitoring didn’t notice. The fix is to measure whole user journeys, such as sign-up to first purchase, rather than whether each server answered.
An average hides the customer who matters
If a few customers bring in most of your revenue, an error rate averaged across everyone can look tolerable while one of them can’t use your product at all. Ten customers burning through their error budget at an average rate of 10 could be one customer at 100 and nine at zero. If you sell to other businesses, watch your biggest customers individually.
One big number hides small broken paths
A single reliability target across a whole service can stay green while one feature, region or type of request is completely broken, because that slice is too small to move the total. Google describes an access proxy whose single 99.99% target missed several outages for exactly this reason. Giving each important path its own check fixed it.
Some failures never reach your servers
DNS faults, load balancer problems and expired SSL certificates all happen before a request reaches your code, so your server graphs stay calm. Automatic retries can hide failures too: the request eventually succeeds, the metric looks fine, and the user waited ten seconds. Server-side numbers can look much better than what users experience, which is why we pair them with checks run from the outside.
Monitoring can break without anyone noticing
If the alert pipeline itself stops working, the result looks exactly like a quiet night. Monitoring shouldn’t depend on the systems it watches, and every setup needs a heartbeat (an alert that fires when expected signals stop arriving). Our alerts guide covers this in more detail.
Why this matters more with AI in the loop
AI coding tools mean teams ship more changes, and more of the code was never read end to end by a person. When something breaks, the context has to be rebuilt from what production can tell you, not from memory.
AI investigation has the same limit. It can only reason about the data it can read. With traces and structured logs that carry the release version, it can connect a spike in errors to the change that caused it and show the evidence. With only CPU and memory graphs, its likely cause is a guess. Better telemetry makes both people and AI faster, and it makes the AI’s answers easier to check.
Where to start
- Keep your tools. You don’t need a new platform. OpenTelemetry (an open standard for collecting metrics, logs and traces) is accepted by most major monitoring tools, so you can add data without migrating. Moving to a new monitoring platform can take more than a year.
- Check one journey end to end. Pick the journey that matters most, such as checkout, and add a synthetic check (a scripted test that runs it every few minutes) that goes all the way through, not just a load of the home page.
- Label every log line and trace. Add the release version and the customer or account identifier, so you can ask “is it one release?” and “is it one customer?” in seconds. Use internal IDs, not names or email addresses, because logs are customer data too.
- Trace that journey. Add tracing to the services behind it, so a slow request shows where it spent its time.
- Add a heartbeat, so a broken alert pipeline can’t pass for a quiet night.
- Count incidents your users found first. Better detection can make it look as though you have more incidents at first, because problems that were always there start showing up.
How we apply this
Cover starts from the alerts you already have, with nothing to install. In the first 90 days we list what your alerts can’t see, including journeys with no end-to-end check, and add a heartbeat. Your monthly report shows how many incidents your users spotted before an alert did, and closing the most important gaps can become one of your quarterly reliability goals. See how we think about monitoring and how our reliability work runs.