Skip to main content

​
Real-World Case Studies

Theory is essential, but seeing how distributed systems work (and fail) in production at massive scale is where deep understanding comes from. This module covers battle-tested architectures from the world’s leading technology companies. Every case study in this chapter follows the same pattern: a company hit a scaling wall, made a set of engineering trade-offs under real constraints (time, money, team expertise, existing code), and lived with the consequences — both intended and unintended. These are not “the right way” to build systems; they are “one way that worked for a specific company at a specific point in time.” The lesson is never “copy Google’s architecture.” It is “understand the reasoning that led to Google’s decisions and apply that reasoning to your own, very different, situation.”
Track Duration: 12-16 hours
Companies Covered: Google, Amazon, Netflix, Uber, Meta, Stripe
Focus: Architecture decisions, failure stories, lessons learned

​
Google Spanner

The database that made the impossible possible — globally consistent transactions. Before Spanner, the conventional wisdom (hardened by the CAP theorem) was that you had to choose between strong consistency and global distribution. Spanner essentially said “what if we throw GPS satellites and atomic clocks at the problem?” and built a system where the laws of physics help enforce transaction ordering.

​
Architecture Overview

​
Key Innovations

The Insight: You can’t synchronize clocks perfectly, but you can bound the uncertainty.
Usage in Transactions:
Per-Tablet Paxos:
  • Each tablet (partition) has its own Paxos group
  • 3-5 replicas across zones
  • Writes go through Paxos leader
  • Reads can go to any replica (with proper timestamp)
Split and Merge:
  • Tablets automatically split when too large
  • Tablets merge when too small
  • Paxos ensures consistent split/merge
The Guarantee: If transaction T1 commits before T2 starts, T1’s timestamp < T2’s timestamp.Why It Matters:
Cost: Commit wait adds latency (~7ms average)

​
Spanner Failure Story: The Leap Second

​
Amazon DynamoDB

The database that powers amazon.com checkout—when availability is everything.

​
Architecture Overview

​
Key Design Decisions

The Origin Story (from the 2007 Dynamo paper):
Implementation:
  • Leaderless replication
  • Write to W of N replicas (W < N means some can be down)
  • Read from R replicas, resolve conflicts
  • Shopping cart uses “union” merge (keep all items)
Problem: Nodes joining/leaving causes massive data reshufflingSolution: Virtual nodes
Original Problem:
  • Provisioned throughput (e.g., 1000 WCU)
  • Uniform distribution assumed
  • Hot partition = throttling
Solution: Adaptive capacity

​
DynamoDB Failure Story: The 2015 US-EAST-1 Outage

​
Netflix: The Chaos Engineering Pioneers

Netflix serves 230+ million subscribers with 99.99% availability. How?

​
Architecture Overview

​
Key Patterns

Scale:
  • 30+ million requests/second
  • 1+ trillion operations/day
  • Petabytes of cached data
Architecture:
Responsibilities:
  • Authentication
  • Dynamic routing
  • Load shedding
  • Request throttling
  • Attack detection
Scale: 1+ million RPS at the edgeInnovation: Zuul 2 (async/non-blocking)
  • Moved from thread-per-request to event loop
  • 90% reduction in connection memory
  • Better tail latency under load
Chaos Monkey (2011): Randomly kills instances in production
Philosophy:

​
Netflix Failure Story: The 2012 Christmas Eve Outage

​
Uber: Real-Time at Scale

Uber processes millions of trips daily with sub-second dispatch decisions.

​
Architecture Evolution

​
Key Systems

Problem: Matching riders to drivers needs consistent, fast routingSolution: Ringpop (swim + consistent hashing)
Challenge:
  • Need horizontal scaling (MySQL doesn’t shard easily)
  • Need flexible schema (trip data evolves fast)
  • Need low latency (real-time dispatch)
Solution: Schemaless (MySQL + application-level sharding)
Uber’s solution for long-running, fault-tolerant workflows.
Open Source: Temporal (Cadence fork) now widely adopted

​
Uber Failure Story: The 2019 Mapping Outage

​
Stripe: Financial Transactions at Scale

When money is involved, correctness is everything.

​
Architecture Principles

​
Key Patterns

How it works:
TTL: Idempotency keys typically expire after 24 hours
Problem: Need to update database AND send event atomically
Problem: Tail latency (p99) is often much worse than medianSolution: Send to multiple replicas, use first response

​
Common Patterns Across Companies

Idempotency Everywhere

All companies: Every mutating operation accepts an idempotency key. Stripe: Idempotency-Key header AWS: ClientRequestToken Google: requestId

Chaos Testing

You don’t know if you’re resilient until you test. Netflix: Chaos Monkey, Chaos Kong Amazon: GameDay exercises Google: DiRT (Disaster Recovery Testing)

Circuit Breakers

Fail fast instead of cascading. Netflix: Hystrix (now Resilience4j) Uber: Custom circuit breakers in every service All use some variant of the pattern.

Observability

You can’t fix what you can’t see. Distributed tracing: Zipkin/Jaeger-style Metrics: RED method (Rate, Errors, Duration) Logs: Structured, correlated by trace ID

​
Key Takeaways

  1. Availability often trumps consistency — Amazon’s shopping cart chose availability. Know when this trade-off is acceptable.
  2. Test failure modes in production — Netflix’s Chaos Engineering isn’t optional, it’s essential.
  3. Build for horizontal scale from day one — Re-architecting a monolith is painful. Design for distribution early.
  4. Invest in your primitives — Google built TrueTime. Uber built Ringpop. Your foundations matter.
  5. Every outage is a learning opportunity — The companies with the best uptime have the best postmortems.