Many production failures can be undone. A bad release can be rolled back, an overloaded service can be given more capacity, and traffic can be moved away from a failing zone. Data failures are different: data lost without a usable backup is gone, and data that has leaked cannot be recalled. Our central rule follows: a backup has no value until you have shown you can restore from it.
Start with three numbers, set by the business
Set three targets for each group of data before you design anything.
| Target | What it means | The question it answers |
|---|---|---|
| RPO (recovery point objective) | How much recent data you can afford to lose | If we restore now, how many minutes or hours of writes are acceptable to lose? |
| RTO (here, restore time objective; often called recovery time objective) | How long a restore may take from end to end, including noticing the problem, deciding to act and finishing the restore | How long can this data be unavailable or wrong? |
| RWO (restore window objective) | How far back in time you can restore | If bad data crept in three weeks ago, can we still reach a clean copy? |
Set these from business need first, then design backups to meet them. A target copied from what your current backups happen to do can never show you a gap. A measurable goal we would start from is that backups exist for 99% of data sources within the agreed RPO, 99% of the time. RWO is the one teams most often forget: if corruption goes unnoticed for a month and you only keep seven days of backups, every copy you hold is already bad.
We also suggest sorting data into a few buckets, each with its own three targets, and giving any data you deliberately do not back up an explicit target too, so the decision is written down.
Replication is not a backup
Many teams run a database with replicas (live copies kept in step with the main one) in another zone or region and assume that covers them. Replication protects you against losing a machine or a location. It does not protect you against a bad write, a faulty migration or an accidental deletion, because those changes are copied to every replica as faithfully as good ones. By the time anyone notices, every live copy is likely to have it.
Snapshots need similar care. They are cheap and fast, but they usually share fate with the live data: if they sit in the same account, region or storage system, the same failure or the same mistaken command can take both. When we say backup, we mean a separate copy that does not depend on the system it protects. For this we recommend the 3-2-1 rule: three copies of the data, on two different types of storage, with one copy kept off-site. For critical data we also recommend point-in-time recovery (restoring to any moment, not just to the last nightly copy).
Keep deletion rights away from live systems
Plan for the ways data actually gets damaged: operator error or deliberate harm, applications writing bad data, bugs in storage software, hardware failure and the loss of a whole region. Count automated tools and AI agents alongside people as actors that can make destructive mistakes, since they act with real permissions and can act quickly. We recommend arranging operations so that no single action, by a person or an AI agent, can wipe out data everywhere at once, and making deletion happen in stages with a way to undo it.
Our rule is that anything able to write to your live data, whether an application, a deployment pipeline, a script or an AI tool, should have no administrative access to your backups. If the credentials that run your service can also delete its backups, one compromised key or one confused automation can remove both at the same time.
Test restores on a schedule, and check what comes back
We treat restore testing as the most important part of any backup plan. Our recommended practice is to restore regularly, ideally automatically, and to alert when a restore fails or takes longer than the agreed RTO. Check the restored data at the application level, not just that a file arrived. Picture a timing bug that quietly produces backups that look complete but are not usable: only a check of the actual contents would catch it.
A useful automated restore test answers three questions: did the restore finish, how long did it take compared with the RTO, and does the restored data make sense to the application (row counts, recent records present, the service able to start against it). A team that cannot find the people, time or budget for a test restore probably cannot find them for a real one either.
Two habits also help: stagger backup schedules so everything does not run at midnight, and make backup setup automatic for every new database so nothing is missed when a team adds one.
Write the recovery policy before you need it
Restoring data often means choosing between options that each lose something, for example rolling back to last night and losing today’s orders, or keeping today’s data and living with some corruption. Those are business decisions, so we recommend writing a data recovery policy in advance and having leadership approve it. The policy should answer who can override it, who approves data-loss decisions outside working hours, whether engineers may look at corrupted data while investigating, and when users will be told that data is declared lost. For one published model, Google has described a policy for its Bigtable database service that puts the integrity of data above speed of recovery and forbids manual changes to data without senior sign-off.
The backup plan we recommend runs in five stages: find and classify your data, set the three targets, design the backups, design how you will validate them, and get formal sign-off from leadership. A spreadsheet of data stores, their targets, where their copies live and when each was last restored is a good first version.
How we apply this
In the first weeks of cover, our gap review checks which databases have backups and a restore that has been tested, and you get the gaps as a written list, so you know what cover can and cannot see. We do not change data in your systems on our own: anything without a runbook you have approved goes to your on-call engineer with the evidence, and the person on your side who makes the call is agreed before cover starts. Drills in our reliability work include rehearsing failures, including the runbooks themselves. See how cover starts.