Most teams write something after a bad outage, but the next outage is often no less likely because of it. Whether a report helps usually comes down to how the incident was handled while it was happening, what the report looks for afterwards, and whether anyone checks that the agreed fixes were done.
Triage and fixing are different jobs
When an alert fires, the first job is triage: deciding how serious it is, who is affected and who needs to be involved. We count triage as successful when it classifies the problem quickly and accurately and gets it to the right people, not when it finds the fix. A fast stand-down on a false alarm counts as a success too. Keeping the two apart in your timeline shows where the time actually went.
We also recommend mitigating before diagnosing. Generic mitigations (fixes that are safe to apply without knowing the cause) such as rolling back the last release, adding capacity, restarting a service or moving traffic away from a failing zone often stop the user pain long before anyone understands what went wrong. Watch for outcome bias here (judging a decision by a result nobody could have known at the time): a rollback that turned out to be unnecessary was still a reasonable call if it was the fastest safe option.
Look for contributing factors
Incidents rarely have a single cause. The urge to find one simple explanation is strong, and in teams where blame is common the quick answer becomes human error. Picking the first event in a chain and calling it the cause is somewhat arbitrary, and a timeline tells you what happened without telling you why.
Contributing factors are the several conditions that together let the incident happen: a missing test, an alert that never fired, a release process that pushed everywhere at once, schedule pressure, a runbook (a written step-by-step fix) that nobody had tried. The test we apply is whether your actions would prevent the whole class of problem as well as this exact bug.
Write it blamelessly
Blameless writing does not mean nobody is accountable. It means the report asks what about the situation made a decision seem sensible at the time. This approach, known as blame awareness, still holds people to normal engineering standards. A related idea, restorative accountability, asks the people closest to the decisions to help find and own the improvements instead of being punished.
In practice, describe each action neutrally, then explain the context: what the dashboards showed, what the runbook said and what else was happening.
What a good report contains
Our default incident report (often called a postmortem) covers the key facts, impact, contributing factors and lessons learned, and adds several sections that are easy to skip.
| Section | What it should answer |
|---|---|
| Timeline | When it started, when it was detected, when people were paged, when it was mitigated and resolved. Gaps between these show where to improve. |
| Impact | Which customers, users or systems were affected, for how long, and whether they had a workaround. |
| Contributing factors | The conditions that together allowed the incident, including process and communication, not only code. |
| What went right | Defences and decisions that limited the damage, so you keep them. |
| Where we got lucky | Things that helped by chance and will not be there next time. |
| Customer communications | Every update sent to customers, when it was sent and whether it was accurate. |
| Follow-up actions | Each action with a named owner, a date and a link to the factor it addresses. |
Have someone outside the incident lead the review
We recommend that the review be led by someone who was not part of the response. That person gathers the chats and documents, interviews the people involved, and guards against outcome bias and hindsight bias (assuming that what is obvious now was knowable then). In a small team, this can be an engineer from a neighbouring service.
Track whether the actions close
A report earns its credibility when its actions close, and that needs named owners and enough team time to do the work. We also recommend setting targets for the report process itself, such as time to first draft and time to publish, so that reports do not drift for weeks while memories fade. To start, put a date on the draft, put a date on each action, and review the list of open actions every month.
Learn from near misses
James Reason’s Swiss cheese model (each defence has holes, and harm gets through only when the holes line up) is the reason we study near misses as well as incidents. An alert where only one safeguard held, or a bad release caught by a single reviewer, shows you a weak layer before it costs anything.
Communication is part of the incident
During a long outage, customers judge the response on two things: how quickly service comes back, and what they are told while it is down. Slow, vague or inaccurate updates leave them unable to decide whether to use a workaround, warn their own users or wait, so poor communication adds harm of its own on top of the outage. It is also the part of a response most likely to be improvised, because the engineers fixing the problem are rarely free to write careful customer updates at the same time.
We recommend treating communication as a role with a named owner. Decide in advance who sends customer updates for any significant incident, and page that person automatically alongside the responders. Agree how often updates go out, write templates and a short communication playbook before you need them, and agree a process for contacting customers directly when the impact is wide. In the report, set each update against the timeline and check whether what it said turned out to be true.
When to go deeper: CAST
For serious or puzzling incidents, we recommend CAST (Causal Analysis based on Systems Theory), a structured way of looking at who or what was meant to keep the system safe, what each of them believed at the time, and which wider factors (such as how changes are managed or how teams communicate) let things drift. It costs more than a standard report. In a side-by-side comparison Google has published on one incident, a standard postmortem took three people 16 hours and produced eight recommendations, while CAST took five people 81 hours and produced 22. It suits the few incidents where a normal report leaves you unsure you understood what happened.
How we apply this
Every incident we handle gets a written report on its contributing factors, posted in Slack and your ticket. Our runbook starter pack offers generic mitigations such as rolling back, adding capacity, restarting or moving traffic, each for your approval, and anything new goes to your engineer with the evidence. Your monthly service report shows which follow-up actions are still open. See how it works