Disaster recovery that works when you need it

Almost every organization has backups. Far fewer know how long a restore takes, whether the restored data is usable, or who would run the process at three in the morning. Disaster recovery plans tend to be written once, approved, and never exercised. The first real test is the disaster.

Recovery is a capability, not a document. It has to be designed per system, built into the platform and practiced on a schedule, like any other operational skill.

Two numbers per system

Two objectives define what recovery means for a system:

• Recovery point objective (RPO): how much data you can afford to lose, measured in time. An RPO of 15 minutes means that after an incident, at most the last 15 minutes of writes may be gone.
• Recovery time objective (RTO): how long the system can be unavailable before the business impact is unacceptable.

These are business decisions, made with the people who own the service. Engineering's job is to say what each target costs. A near-zero RPO with a few minutes of RTO needs continuous replication to a standby that is always running. A 24-hour RPO with a one-day RTO needs a nightly backup and a tested runbook.

Most estates settle into a few tiers:

Assign every system to a tier and write the tier into its service catalog entry. A system without an assigned tier has, in practice, an RTO of "whenever someone figures it out".

Backups and replication solve different problems

Replication and backup are often treated as interchangeable. They protect against different failures.

Replication keeps a second copy current. Aurora Global Database, DynamoDB global tables and S3 Cross-Region Replication give you a low RPO when a region or an availability zone fails. But replication copies everything faithfully, including mistakes. A bad migration that drops a column, a script that deletes the wrong rows, or ransomware encrypting objects are all replicated within seconds.

Backups are point-in-time copies kept separately. They let you go back to before the mistake. They are slower to restore, and that is the trade.

Tier 1 systems need both. Replication handles infrastructure failure. Backups handle data corruption and human error, which in practice are the more common disasters.

Encode the plan, not just the schedule

Backup schedules belong in infrastructure as code, next to the resources they protect. With AWS Backup, a plan selects resources by tag, so every database tagged with a tier is protected without anyone remembering to add it:

resource "aws_backup_plan" "tier2" {
  name = "tier2"
 
  rule {
    rule_name         = "hourly"
    target_vault_name = aws_backup_vault.primary.name
    schedule          = "cron(0 * * * ? *)"
 
    lifecycle {
      delete_after = 35
    }
 
    copy_action {
      destination_vault_arn = "arn:aws:backup:eu-central-1:210987654321:backup-vault:dr-tier2"
      lifecycle {
        delete_after = 35
      }
    }
  }
}
 
resource "aws_backup_selection" "tier2" {
  name         = "tier2-resources"
  plan_id      = aws_backup_plan.tier2.id
  iam_role_arn = aws_iam_role.backup.arn
 
  selection_tag {
    type  = "STRINGEQUALS"
    key   = "dr-tier"
    value = "2"
  }
}

The same pattern applies outside AWS: Azure Backup policies and Google Cloud Backup and DR both attach schedules and retention to resources through policy, so coverage follows tags rather than memory.

Do not forget what is not a database. Secrets, DNS zones, IAM configuration, container images and the infrastructure code itself are all needed to recover. If the only copy of the Terraform state lives in the region that just failed, rebuilding will be slow. The approach in Structuring Terraform for many environments makes it easier to stand up the same stack in a second region.

Test restores on a schedule

A backup you have never restored is a hope, not a backup. Restore testing should be automated and boring:

1. Restore the latest recovery point into an isolated account or VPC.
2. Run checks against the restored copy: the database starts, key tables exist, row counts are within expected ranges, the most recent records are newer than the RPO allows.
3. Record how long the restore took. That number is your measured RTO for the data layer.
4. Delete the restored resources.

AWS Backup restore testing runs steps 1 and 4 on a schedule. For step 2, a Lambda function triggered by the restore job's completion event can run your checks and report the outcome back with the PutRestoreValidationResult API. On other platforms, a scheduled pipeline that restores, validates and tears down does the same job.

A minimal validation for a restored PostgreSQL database might look like this:

#!/usr/bin/env bash
set -euo pipefail
 
latest=$(psql "$RESTORED_DB_URL" -Atc \
  "select extract(epoch from now() - max(created_at)) from orders")
count=$(psql "$RESTORED_DB_URL" -Atc "select count(*) from orders")
 
if (( ${latest%.*} > 3600 )); then
  echo "restore is ${latest%.*}s old, exceeds 1h RPO" >&2
  exit 1
fi
if (( count < 1000000 )); then
  echo "orders table has only $count rows" >&2
  exit 1
fi
echo "restore ok: $count orders, newest is ${latest%.*}s old"

Alert when a test fails or does not run. A silent restore job that stopped three months ago is a common finding in recovery reviews.

Game days

Automated restores prove the data comes back. They do not prove the organization can recover a service. Game days do.

A game day is a planned exercise where a team simulates a failure and recovers from it using only the runbook. Start small and in a non-production environment: restore one database and point a staging service at it. Over time, work up to failing over a tier 1 service to its standby region.

During the exercise, note every point where someone had to ask a question, look something up outside the runbook or wait for access. Those are the gaps a real incident will hit. Typical findings include missing permissions in the recovery account, DNS records with long TTLs, hard-coded regional endpoints in configuration, and runbook steps that refer to a console page that has since moved.

Measure the elapsed time from the start of the exercise to a working service and compare it with the RTO. If they differ, either the design or the objective needs to change, and that conversation is far easier to have after a game day than after an outage. Tools such as AWS Fault Injection Service can make the failure realistic, for instance by disrupting connectivity to a subnet, once the basic recovery path is reliable.

Where to start

• List your systems and agree on a tier, RPO and RTO for each with the owning team.
• Check that every tier 1 and 2 system has both replication or point-in-time recovery and backups copied to another account and region.
• Manage backup plans as code and select resources by tag.
• Automate a weekly restore test for each critical data store, with validation and an alert on failure.
• Run one game day per quarter, starting with a single restore, and fix every gap it finds before the next one.