Skip to main content

High Availability & Disaster Recovery

Design systems that survive failures and disasters. Learn to achieve 99.99% availability. Azure HA/DR Patterns

What You’ll Learn

By the end of this chapter, you’ll understand:
  • What High Availability and Disaster Recovery really mean (and why they’re different)
  • How to calculate and achieve specific SLAs (99.9%, 99.99%, 99.999%)
  • What RPO and RTO mean (and why confusing them costs companies millions)
  • How to design systems that survive datacenter failures (Availability Zones)
  • How to design systems that survive regional disasters (Multi-region DR)
  • Real-world DR architectures with actual costs and trade-offs
  • How to test your DR plan (because untested plans always fail)

Introduction: What is High Availability & Disaster Recovery?

Start Here if You’re Completely New

High Availability (HA) = Your app stays online even when things break Disaster Recovery (DR) = Your app can recover from catastrophic failures Think of it like a restaurant: High Availability (HA):
  • Problem: One cook gets sick
  • Solution: You have 3 cooks (redundancy)
  • Result: Restaurant stays open ✅
  • Downtime: 0 seconds
Disaster Recovery (DR):
  • Problem: Fire destroys the entire restaurant
  • Solution: You have a second location across town (backup site)
  • Result: Open at backup location in 2 hours ✅
  • Downtime: 2 hours (but you survived!)
Key Difference:
  • HA = Handles small failures (broken VM, network glitch) → Seconds of downtime
  • DR = Handles catastrophic failures (datacenter destroyed, region offline) → Hours of downtime

Why This Matters: The Cost of Downtime

Real-World Disaster Example

Amazon Prime Day Outage (2018)
  • What happened: Website crashed for 63 minutes during biggest sale day
  • Revenue loss: $99 million in 63 minutes
  • Per-minute cost: $1.57 million/minute
  • Per-second cost: $26,000/second
Every second your app is down = money lost + customers lost + reputation damaged. More Real Examples: The Pattern: Companies that invest in HA/DR save millions. Companies that don’t, lose millions.

Understanding SLAs (Service Level Agreements)

What is an SLA?

SLA = A promise about how much downtime is acceptable Think of it like a pizza delivery guarantee:
  • Pizza delivery SLA: “30 minutes or it’s free”
  • Azure VM SLA: “99.9% uptime or we give you credits”

SLA Math Explained (From Scratch)

99.9% uptime sounds amazing, right? Let’s see what it actually means:
Translation Table: Example: Your e-commerce site makes $100,000/day
  • 99.9% SLA: 43 minutes downtime/month = $3,000 lost revenue
  • 99.99% SLA: 4.3 minutes downtime/month = $300 lost revenue
  • Cost to upgrade: ~$200/month
  • Savings: $2,700/month → 13.5x ROI ✅

How Azure Achieves High Availability

The Building Blocks (From Smallest to Largest)

Let’s understand each one:

1. Single VM (No High Availability)

Real-World Analogy: Running a restaurant with only 1 cook. Cook gets sick = restaurant closes.

2. Availability Set (Same Datacenter, Different Racks)

Real-World Analogy: Restaurant with 3 cooks. One cook gets sick = other 2 keep working. Cost Example:
  • 1 VM (99.9%): $50/month → 43 min downtime
  • 3 VMs in Availability Set (99.95%): $150/month → 21 min downtime
  • Extra cost: $100/month → Saves 22 minutes of downtime

3. Availability Zones (Different Datacenters, Same Region)

Real-World Analogy: Restaurant chain with 3 locations in same city. One location catches fire = other 2 still serve customers. Cost Example:
  • 3 VMs in Availability Set (99.95%): $150/month → 21 min downtime
  • 3 VMs across Availability Zones (99.99%): $150/month → 4.3 min downtime
  • Extra cost: $0 (same price!) → Saves 17 minutes of downtime ✅
Why Availability Zones are Better:
  • Protects against datacenter-level disasters (fire, flood, power outage)
  • Same cost as Availability Set (this is the key insight — same price, better SLA)
  • Higher SLA (99.99% vs 99.95%)
  • No additional configuration complexity compared to Availability Sets
Practical Tip: Always default to Availability Zones for new deployments. The only reason to use Availability Sets today is if your Azure region does not support Availability Zones (check Azure’s region documentation) or if you are working with legacy services that do not yet support zone-redundant deployments. As of 2025, most commonly used services (VMs, Azure SQL, AKS, App Service) fully support Availability Zones. Cost Consideration: While the VMs themselves cost the same across zones, data transfer between Availability Zones within a region costs 0.01/GB.Formostapplications,thisisnegligible(0.01/GB. For most applications, this is negligible (10/month for 1 TB of cross-zone traffic). However, for data-intensive workloads (database replication, large file transfers), factor this into your cost model. It is still dramatically cheaper than multi-region replication ($0.05-0.12/GB).

4. Multi-Region (Disaster Recovery)

Real-World Analogy: Restaurant chain with locations in New York and London. Hurricane destroys New York = London location still serves customers. Cost Example:
  • Single Region (East US): $150/month → 4.3 min downtime
  • Multi-Region (East US + West Europe): $300/month → 2 min downtime
  • Extra cost: $150/month → Protects against regional disasters
When You Need Multi-Region:
  • ✅ Mission-critical applications (banking, healthcare)
  • ✅ Global user base (low latency everywhere)
  • ✅ Compliance requirements (data residency)
  • ❌ Small internal tools (not worth the cost)

1. Availability SLAs


Understanding RPO & RTO (From Absolute Zero)

What is RPO and RTO?

These are the TWO most important numbers in disaster recovery. Companies have lost millions by confusing these. RPO (Recovery Point Objective) = “How much data can we afford to lose?” RTO (Recovery Time Objective) = “How long can we be offline?”

Real-World Analogy: Writing a Book

Imagine you’re writing a 500-page book on your computer: Scenario 1: You save every 5 minutes (RPO = 5 minutes)
Scenario 2: You save every 1 hour (RPO = 1 hour)
RPO = Time between backups = Amount of data you can lose

Now RTO (Recovery Time Objective)

RTO = How long until you’re back to work after a disaster Continuing the book analogy: Scenario 1: Backup laptop ready (RTO = 5 minutes)
Scenario 2: Need to buy new laptop (RTO = 3 days)
RTO = Time to recover from disaster = How long you’re offline

The Critical Difference (Why Companies Confuse This)

[!WARNING] Common Mistake: Confusing RPO and RTO RPO = Data Loss (measured in TIME since last backup) RTO = Downtime (measured in TIME to recover)
You can have DIFFERENT combinations: Example 1: Low RPO, High RTO
Example 2: High RPO, Low RTO
Example 3: Low RPO, Low RTO (Expensive but Best)

Real-World RPO/RTO Example: GitLab Database Incident (2017)

The Disaster:
  • GitLab engineer accidentally deleted production database
  • 300 GB of data vanished
What They THOUGHT Their RPO Was: 24 hours (daily backups) What Their RPO ACTUALLY Was: 6 hours (daily backups were failing, only staging backups worked) Actual Result:
  • RPO: 6 hours → Lost 6 hours of data (5,000 projects, 5,000 comments, 700 new users)
  • RTO: 18 hours → Took 18 hours to restore from backups
  • Total impact: 6 hours of data lost + 18 hours offline
  • Cost: Immeasurable reputation damage (but they recovered with transparency)
Lesson: Your DR plan is only as good as your last successful restore TEST.

How to Choose Your RPO/RTO

Step 1: Calculate Cost of Downtime
Step 2: Calculate Acceptable Loss
Step 3: Calculate Cost of DR Solution

Decision Tree: Choosing RPO/RTO


2. RPO & RTO

[!WARNING] Gotcha: RPO vs RTO A common interview trap. RPO (Point) = Data Loss (How far back do we go?) RTO (Time) = Downtime (How long until we are back online?) You can have low RPO (0 data loss) but high RTO (took 4 hours to restart).
[!TIP] Jargon Alert: Split Brain A disaster scenario where two databases both think they are “Primary” and accept writes at the same time, corrupting data. Always use a “Witness” or “Quorum” to preventing this in active-active architectures.
Quick Reference (after reading the detailed explanation above): RPO (Recovery Point Objective): How much data loss is acceptable? RTO (Recovery Time Objective): How long to recover?

Example Scenarios


Common Mistakes in HA/DR (Learn from Others’ Failures)

Mistake #1: “Stopped” VMs Still Cost Money

The Trap:
What Happened:
  • “Stopped” = OS shutdown, but VM resources still reserved
  • You still pay for compute, just not OS license
  • Correct action: “Deallocate” (not just “Stop”)
Cost Impact:
  • Stopped VM: Still ~80% of full cost
  • Deallocated VM: Only pay for storage (~5% of full cost)
Fix:
Savings: $1,600/month per VM ✅

Mistake #2: Untested Backups (The GitLab Disaster)

The Trap:
Real Example: Code Spaces (2014):
  • Hosting company for developers
  • Backups existed but were on same infrastructure
  • Hacker deleted everything (including backups)
  • Company went out of business
  • Customers lost everything
The Fix: Test quarterly
Cost of Testing: $500/month (test infrastructure) Cost of Untested Backup: Business bankruptcy ❌

Mistake #3: Ignoring Composite SLAs

The Trap:
The Math (Explained Simply):
Real Example:
The Fix: Add redundancy

Mistake #4: Forgetting About Data Gravity

The Trap:
Data Gravity = Large data is slow to move Real Numbers:
The Fix: Plan for data size

Mistake #5: Active-Active Without Proper Conflict Resolution

The Trap:
Real Example: Uber (2016):
  • Active-active setup without proper conflict resolution
  • Network partition between datacenters
  • Both sides accepted writes
  • Data corruption cost hundreds of hours to resolve
The Fix: Choose conflict resolution strategy

3. High Availability Patterns


4. Disaster Recovery Architecture

Azure Site Recovery Multi-Region DR Architecture

Example: E-Commerce Platform


5. Backup Strategies


6. Testing DR Plan

Disaster Recovery Drill Checklist:


7. Interview Questions

Beginner Level

Answer:
  • Availability: Uptime. Can I access the service right now? (e.g., SLA 99.9%).
  • Durability: Data integrity. Is my data safe from loss? (e.g., 11 nines 99.999999999% for Blob Storage). You can have high availability but lose data (corruption), or high durability but be offline.
Answer: A physically separate datacenter within the same Azure Context (Region). It has independent power, cooling, and networking. Protects against datacenter-level failures (fire, power cut).

Intermediate Level

Answer:
  • RPO (Recovery Point Objective): “How much data can we lose?” (Time since last backup).
  • RTO (Recovery Time Objective): “How long can we be down?” (Time to restore service).
Answer:
  • Active-Passive: One region handles traffic. Secondary is standby. Cheaper, slower failover (RTO > 0).
  • Active-Active: Both regions handle traffic. Complex data sync. Expensive. zero downtime failover (RTO ≈ 0).

Advanced Level

Answer: By using Composite SLAs. If you have two regions, each with 99.9% availability, the probability of both failing simultaneously is 0.1%×0.1%=0.01%0.1\% \times 0.1\% = 0.01\%. Total Availability = 100%0.01%=99.99%100\% - 0.01\% = 99.99\%. Redundancy increases availability.

8. Key Takeaways

SLA Mathematics

Understand how SLAs compound. Dependencies reduce availability; Redundancy increases it.

Zones vs Regions

Use Zones for synchronous HA (High Availability). Use Regions for asynchronous DR (Disaster Recovery).

Data Gravity

Compute is stateless and easy to move. Data is heavy and hard to sync. Focus DR efforts on data replication.

Testing

A backup is useless if you can’t restore it. A DR plan is a hypothesis until tested.

Business Alignment

RPO/RTO are business decisions, not technical ones. They dictate the cost of the designated solution.

Next Steps

Continue to Chapter 14

Master real-world Azure architecture patterns and design principles