How to write your first runbooks, and when to let them run on their own.

In most young engineering teams, the fix for a recurring problem lives in one or two people’s heads. When they are asleep, on leave or have moved on, the next person paged has to work it out again from scratch, usually at night and under pressure. A runbook (also called a playbook: a written, step-by-step fix for a problem you have seen before) is the cheapest way to change that.

Stop the harm first, then find the cause

Our rule during an incident is that mitigation (reducing the harm to users) comes before diagnosis (working out why it happened). Users are affected until the harm stops, and a full diagnosis can take hours. We start with five generic mitigations: actions that are safe to take before anyone knows the cause because they help across many kinds of failure. They make good first runbooks because almost every service needs them sooner or later.

One caution: these mitigations are themselves changes to production. We treat them with the same care as any other change, which in practice means doing them gradually and watching the result, rather than draining every zone at once.

Start with your five most recent outages

For a small team that is busy firefighting, our default starting point is to write runbooks for your five most recent outages. These are failures you know happen, so prepare for them before ones you can only imagine.

Write each one for the person most likely to need it: a developer or operations engineer who is not a reliability specialist and has been paged at 2am. In practice that means:

• a plain description of what the alert or symptom looks like, so the reader can tell whether this runbook applies;
• the checks to run first, with the exact commands or dashboard links;
• the fix, one action per step, with what “working” should look like after each step;
• how to undo the fix if it makes things worse;
• who to escalate to, and what to tell them.

Even if the runbook only covers initial troubleshooting, escalations start arriving with the basic checks already done. We also recommend keeping runbooks current through small, fast edits after each use.

Keep critical runbooks reachable when the service is down

A runbook stored only on the platform that has just failed is no use during that failure. We recommend keeping critical runbooks somewhere independent of the service, such as a code repository, a local bundle of essential documents (sometimes called a panic package) or offline copies, so they can be reached through more than one route. Check where yours live, and whether someone could open them if your single sign-on (the one login for all your internal tools) or internal wiki were unavailable.

Test and practise before an incident relies on them

A team can believe it is able to roll back and only discover during an outage that it cannot. An untried runbook is a guess, so we recommend practising mitigations in advance and running realistic tests that are not optional. A simple routine is to try each new runbook in a staging environment (a copy of production used for testing), then rehearse the most important ones in a short drill each quarter, recording the date each one was last exercised.

Automate in stages

We treat every runbook as a rough draft for later automation, and we take it one step at a time: watch the task done, do it by hand yourself, script it with a person triggering the final step, then automate it fully. Doing it by hand first is how you learn the edge cases the script has to handle. Our default is that new automation climbs a trust ladder:

1. Detect and alert. The automation spots the situation and tells an engineer, who checks whether it was right.
2. Suggest. It proposes the action and a person reviews it before anything runs.
3. Act and notify. It runs the action itself and tells people what it did.

Powerful automation also needs safety features. Scope settings (which account, region or service it may touch) should be required and should fail safe: if a setting such as the region is missing, the automation should act on nothing, never on everything, because a missing value is far more likely to be a mistake than an instruction. Add limits on how much it can touch in one run, such as a maximum share of capacity it may drain; checks before acting, so it does no harm; a circuit breaker that halts and escalates to a person when something looks wrong; and a written, tested undo step.

We keep people on the critical decisions because automation only sees its inputs, and those inputs can be wrong. Google has published an example in which a monitoring fault set off every alert at once, and a fully automated recovery would have restored a database that did not need it and lost recent user data.

Watch for runbooks that run more and more often

Take automatic restarts for a memory leak (a program slowly using up memory until it fails). Restarting is acceptable if the leak is rare and expensive to fix, but once restarts become far more frequent than releases, moving from daily to hourly, the automation is hiding a problem that should be fixed and quietly building up technical debt (shortcuts that cost more to fix the longer they stay).

How we apply this

In the first 90 days we read your five most recent incidents and, by day 30, offer a starter pack of standard fixes such as rolling back a release, adding capacity, restarting or moving traffic away from a failing zone, each tried before it can run in a real incident. A runbook runs only for a known issue you have approved it for, after an engineer confirms the cause, every run is logged, and one action in Slack or the portal pauses them all. Our drills rehearse the runbooks themselves, and our on-call coaching helps your engineers write runbooks that make sense at 2am. You can read more on how you keep control of production.