Imagine a team accidentally wiping its production database. Backups exist, but restoring them takes longer than the business can tolerate. Without agreed recovery targets and a practiced playbook, having backups alone does not make the team ready.
A disaster recovery plan connects business recovery objectives to backup protection, dependency order, executable runbooks and evidence from drills. This guide distinguishes RTO from RPO, compares standby approaches and explains how to validate recovery without treating an available backup as a complete plan. The numerical examples and schedules are illustrative, not promises for a particular system.
Turn recovery objectives into a tested plan
A written plan becomes useful when authorized people can follow it with the access, infrastructure and data available during the intended failure. DR is part of business continuity: a restored checkout application may still be unusable if identity, payments or fulfillment cannot operate. Define the minimum acceptable service, its dependencies and the evidence needed to declare recovery.
Record who coordinates the incident, who changes systems and who communicates with stakeholders. A shared process reduces conflicting changes and makes decisions traceable. Start with business impact and recovery objectives, then choose the technical measures and budget needed to pursue them.
RTO and RPO: objectives and measured results
Recovery Time Objective (RTO) is the maximum acceptable interval from service interruption to restoration at the agreed usable service level. Recovery Point Objective (RPO) is the acceptable data-loss window between the last recoverable consistent state and the interruption. Both are objectives; actual recovery duration and recovered-data coverage are results measured against them.
For example, suppose an interruption starts at 10:00, the last recoverable consistent state is 09:56, and the agreed service checks pass at 11:00. Measured recovery takes 60 minutes and the data-loss window is 4 minutes. With targets of RTO 90 minutes and RPO 5 minutes, this particular drill meets both. RPO is not measured backward from 11:00. If corruption began earlier, identify the actual last known good point and its loss window separately.
Assign objectives per service from business impact, including time-sensitive operations such as payroll or a campaign. Budget the complete interruption: detection, decision, access, provisioning, data restoration, application verification and traffic transition. Logging needed for diagnosis, security and verification remains a dependency even when checkout has the highest business priority.
A hypothetical business might choose a five-minute RPO for orders and two hours for product images if that matches its needs. Different data policies still need a coherent recovery: the restored application must handle any missing objects or references correctly. Test the whole customer path rather than optimizing each backup in isolation.
Map dependencies and failure scenarios
Before you write your first runbook, you need a clean map. Don’t worry, it doesn’t need to be pretty. It just needs to be honest. Start with the customer‑facing paths that make or save money. From the moment a user lands to the moment they pay, what services are in that chain? Web tier, API layer, database, cache, object storage, payments, DNS—walk it end to end. Then ask, “What does this depend on?” And keep asking until you hit the bottom of the stack: network, identity, keys, logging, alerts.
Include less visible dependencies: DNS caches, shared Redis sessions, object storage, certificates, deployment artifacts and secret stores. A restored database cannot serve the application if its credentials are trapped in an unavailable region. A simple map showing which service calls which dependency is useful when verified against configuration and service owners.
Choose scenarios from impact and likelihood: region loss, accidental deletion, corruption, compromised credentials or an unavailable critical vendor. These require different recovery paths. A healthy replica may help with a host outage but may already contain the same deletion as the primary. For suspected compromise, preserve relevant incident evidence and select a verified clean recovery point before reconnecting systems.
Set a dependency-aware recovery order. For example, restore shared network, identity and database prerequisites before enabling customer login or checkout, then restore lower-priority services as their objectives allow. Record the decision owner and the conditions that activate DR early enough to complete the recovery steps; do not wait until the RTO has already elapsed.
Protect recovery points and test the restore
Match data capture and transfer to the RPO, and the complete recovery path to the RTO. A successful backup schedule alone is insufficient: check the last usable consistent point, required logs, access and restore duration. Replication can propagate deletion or corruption, so retain independently protected past recovery points even with a standby.
A database recovery method may use a consistent full backup, dependent incrementals and supported WAL or binary-log replay. Incremental backups do not inherently allow an arbitrary second as a recovery target. Follow the engine, version and tool’s procedure, including external configuration and a complete required log chain. Snapshots must satisfy the application/database consistency requirements; an ordinary live-directory copy is not a universal substitute. Account for restore capacity and service quotas during a regional event.
The 3-2-1 guideline means three copies including production, on two media types, with one offsite. Offsite placement and deletion isolation are different properties. A second region under the same unrestricted credentials may still be vulnerable to the same destructive action. Consider offline or immutable recovery copies and separate the ability to write new backups from the ability to delete retained ones.
Encrypt transfers and retained backups, and preserve access to every key still required to decrypt retained data. Key rotation must not make old backups unusable. Record engine/tool version, backup ID, completion time, source and log coverage. Set retention from recovery needs and applicable obligations; validate the oldest required restore point before reducing coverage.
Restore into an isolated environment with production credentials and outbound integrations disabled or replaced appropriately. Verify files, database relationships and business totals against a baseline for the same recovery point; hashes or row counts alone do not prove the application works. Use test accounts and payment sandbox endpoints, and prevent real emails, webhooks or charges. Record elapsed time and blockers, then retest corrections.
The AWS disaster recovery guide compares recovery strategies. From the retained NIST SP 800-34 publication record, choose Revision 1 updated on 11 November 2010; it is federal information-system planning guidance to adapt to your context. The Google SRE incident-management chapter covers roles and coordination. These frameworks do not replace your own recovery tests.
A practical recovery runbook template
A runbook covers one explicit scenario and the actions needed to recover from it. Use a concise structure with prerequisites, target identifiers, steps, expected results and failure branches. Have another authorized operator follow it in a supervised drill to reveal undocumented assumptions.
What goes in the header
Name the action, owner, repository/configuration version, review date and supported RTO/RPO. Identify the affected service, recovery account/region and selected backup or target point. A recorded log-coverage boundary such as 10:15 UTC is an input to verify, not permission to assume later transactions can be recovered.
Pre-checks that save you hours
Check access to the backup, keys, deployment artifacts, destination network and required capacity. Confirm incident scope and who can activate the runbook. For a single-writer design, prevent the old primary from accepting conflicting writes before opening the replacement. If the required log chain or clean recovery point is missing, take the documented data-loss assessment path rather than proceeding with an unverified target.
Steps that flow like a conversation
Each step should name its target and observable result: provision a fresh isolated destination, restore the selected consistent backup using its supported procedure, replay required logs to the verified target, then inspect recovery completion and business data. Compare order counts only with a baseline for that same point; a generic Prometheus snapshot does not automatically contain the relevant order data. Record IDs, timestamps and evidence in the incident log.
DNS changes do not immediately replace every cached answer or close existing client connections. Lowering TTL before a planned transition can help; lowering it during the incident does not retroactively change the TTL of an already cached answer. Verify the destination certificate, capacity and health before routing users. Reuse deployment health checks where useful, while separately addressing recovered data and failed dependencies.
Verification, rollback, and the first hour after recovery
Define meaningful user-level checks, such as test login and order processing, with expected data outcomes as well as response codes. Monitor errors, latency, queues and capacity against the agreed degraded-service threshold. Once Region B accepts writes, returning to an older Region A snapshot can discard them. Failback requires A to be usable, B’s new writes to be preserved and reconciled, and conflicting writers to be prevented. Keep the recovered service running if the original region cannot safely resume.
A small but mighty annex: contacts, tools, and credentials
Keep vendor contacts, decision owners and access procedures in a protected annex; reference the vault rather than copying secrets into runbooks. Test the break-glass route when normal SSO or the primary region is unavailable. Maintain an accessible plan copy and an alternative coordination channel; emergency credentials behind the same failed service are not an independent recovery path.
Choose standby readiness from business needs
In cloud terminology, backup and restore provisions recovery resources when needed. A pilot light keeps core components ready but requires activation or additional deployment to serve requests. Warm standby is a scaled-down, functional environment that can handle limited traffic before scaling. Hot standby keeps a more fully prepared passive destination; active/active serves traffic in multiple locations. These are distinct arrangements, and no label alone proves a particular RTO or RPO.
Choose readiness from each business path’s objectives and dependencies, then test destination capacity, quotas and control-plane availability. A revenue service may justify more standby resources; an internal service may also be critical if recovery depends on it. A hypothetical mix of hot checkout, warm search and cold administration is appropriate only if the impact analysis and drills support it.
High availability can handle some failures without a full recovery, but replicated nodes do not replace independent backups for deletion or corruption. The related comparison of MariaDB Galera and MySQL Group Replication discusses database availability; keep DR recovery-point selection and validation as separate concerns.
Activation, communication and ownership
Assign an incident commander to coordinate decisions, authorized operators to change systems and an owner for communications and the incident timeline. Small teams may combine roles, but the responsibilities still need to be explicit. Record commands, target changes, assumptions and handoffs so the next operator can understand the current state.
Activation triggers should leave enough time to execute DR before the objective is missed. Examples include a forecast that normal recovery cannot finish in time, regional unavailability or confirmed data corruption requiring a known-good point. Define measurable thresholds and decision authority in advance. An integrity or security incident may require action even while endpoints still return successful responses.
Communicate verified impact and current actions, with a time for the next update. An example such as “Restore is in progress; next update in 30 minutes” sets an update commitment without inventing a recovery completion time. Keep alternate internal channels ready if chat or identity systems are part of the failure.
After the dust settles, run a blameless review. Ask: what surprised us, what slowed us, what will we change in the plan? Assign owners and deadlines to the findings—runbook updates, TTL adjustments where justified, or new monitoring checks—and schedule the next drill while the lessons are fresh.
Connect the plan to its automation
Keep a one-page index naming protected services, objectives, activation conditions, scenarios and their runbooks. Link the dependency map, access annex and latest drill results. Where several runbooks are required, state their order and the evidence needed before advancing to the next step.
Each runbook should include the exact working directory, repository revision, account/region, state or inventory, required access, expected changes, verification and rollback/failback conditions. Define who declares the agreed service level restored and who watches the following period. Store secrets in the approved secret system and document its tested recovery path.
The following are CLI examples for an already prepared repository, not a supplied DR implementation. Assume an initialized Terraform root module at dr/region-b with the intended DR account, region, backend state and workspace selected. Review the saved plan and its resource targets before running the apply step separately. A saved plan may contain secrets, and applying it does not ask for another confirmation.
# Example repository paths; initialized module and DR state/account required.
terraform -chdir=dr/region-b plan -out=dr.tfplan
terraform -chdir=dr/region-b show dr.tfplan
# Run separately only after reviewing this saved plan and its target.
terraform -chdir=dr/region-b apply dr.tfplan
For an existing site-dr.yml playbook and inventory/dr-b.yml inventory, run from the repository root. Validate the actual hosts, tasks, variables and access first. Syntax checking and listing hosts do not prove that recovery will succeed; the final command executes the playbook and changes its targets.
# Existing playbook and inventory; run from the repository root.
ansible-playbook -i inventory/dr-b.yml site-dr.yml --syntax-check
ansible-playbook -i inventory/dr-b.yml site-dr.yml --list-hosts
# Execute separately only after validating targets, tasks and access.
ansible-playbook -i inventory/dr-b.yml site-dr.yml
Keep the plan index, automation revision, inventory names and contact/access annex current together. A newly authorized operator should be able to locate and rehearse the correct path without relying on one person’s memory. Record any missing command argument or permission as a concrete follow-up.
A consistent schedule for recovery exercises
An illustrative starting schedule is a small file/database restore each week, a full service recovery each month and a scenario drill each quarter. Choose the actual frequency from criticality, change rate and applicable obligations. Repeat affected recovery tests after major engine, infrastructure, access or retention changes. This is a suggested planning rhythm, not a universal standard requirement.
For a scenario drill, assume the primary region or usual tool is unavailable and use the documented alternative. Include access, routing, data and application checks, not just a restored database endpoint. Start in an isolated environment. A production exercise needs an agreed scope, authorized execution, impact limits, stop conditions and recovery steps; staging alone cannot validate every production dependency.
Track interruption start, detection and decision times, restore completion, user-level acceptance and the last consistent recovered point. Compare the total elapsed time with RTO and the data-loss window with RPO. A first successful response is a milestone, not necessarily full recovery. Assign owners and deadlines to missed objectives and retest the affected path.
Protect secrets and recovery evidence
Apply the required logging, access and network controls to the recovery environment before opening traffic. Predesign emergency access; any necessary exception needs an owner, limited scope/duration, a recorded use and removal afterward. A regional secret store, certificate dependency or inaccessible encryption key must have a tested recovery path. Credential rotation should preserve access to retained encrypted backups.
Record retention decisions, encryption/key access, authorized operators and restore-test evidence according to your organization’s obligations. A generic DR checklist does not establish regulatory compliance. After service recovery, create a new usable backup of the working state, remove temporary access and update the runbook with the final location and configuration.
Check the remaining recovery gaps
Confirm that backup freshness meets the objective and that the selected engine/tool versions can restore it. Verify offsite data and keys without access to the primary region, and keep emergency access independent of a failed identity path. Validate user-level outcomes and preserved post-failover writes before closing the incident. Provider SLA terms alone do not guarantee application recovery across all dependencies.
Use a starter project to make the plan concrete: first map systems and objectives, then validate backup/access coverage, then rehearse the highest-priority runbooks. A three-week outline may suit a small scope, but completion depends on passed acceptance checks and resolved findings rather than the calendar. Review the plan whenever relevant architecture or ownership changes.
Evidence that the plan is ready
A usable plan has owned recovery objectives, a verified dependency map, protected recovery data and keys, explicit runbooks and recent drill records. If a target remains unmet, document the gap, interim measure and retest date. The next exercise should show whether the correction improved the actual recovery path.
