> ## Documentation Index > Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt > Use this file to discover all available pages before exploring further. # Distributed Systems Mastery > Crack FAANG-level interviews with deep distributed systems expertise — Consensus, Raft, Paxos, and production patterns # Distributed Systems Mastery A comprehensive, interview-focused curriculum designed for engineers targeting **Staff/Principal roles at top tech companies** (Google, Meta, Amazon, Netflix, Stripe, etc.). This course covers everything from fundamentals to cutting-edge distributed systems concepts. Distributed systems sit at the intersection of computer science theory (impossibility results, consensus proofs) and messy real-world practice (network jitter, clock skew, disk failures at 3 AM). This curriculum bridges both worlds -- giving you the theoretical foundations to reason about correctness and the production battle scars to design systems that actually work under fire. **Course Duration**: 18-24 weeks (self-paced)\ **Target Outcome**: Staff+ Engineer at FAANG / Top-tier distributed systems expertise\ **Prerequisites**: Strong programming, basic networking, database fundamentals\ **Language**: Concepts with implementations in Go/Java/Python\ **New Content**: 5 additional tracks with 45+ modules, real-world case studies, and Staff+ interview problems *** ## Why This Course? Covers exact topics asked at Google, Amazon, Meta, and other top companies Understand Raft, Paxos, ZAB, and other consensus protocols inside-out Patterns from systems handling millions of QPS at scale Build your own distributed KV store, consensus implementation, and more **Interview Reality**: At Staff+ level, you're expected to design systems that handle **billions of requests**, survive **data center failures**, and maintain **consistency guarantees**. This course prepares you for exactly that. *** ## What Companies Ask | Company | Common Topics | | ----------- | --------------------------------------------------------------------------------- | | **Google** | Consensus protocols, Spanner, Bigtable internals, Paxos, distributed transactions | | **Amazon** | DynamoDB internals, eventual consistency, vector clocks, Dynamo paper | | **Meta** | TAO, ZippyDB, consensus at scale, social graph distribution | | **Netflix** | EVCache, Cassandra, chaos engineering, resilience patterns | | **Stripe** | Distributed transactions, exactly-once delivery, idempotency | | **Uber** | Ringpop, consistent hashing, real-time systems, Cadence workflows | *** ## Course Structure The curriculum is organized into **9 tracks** progressing from fundamentals to Staff+ expertise: ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ DISTRIBUTED SYSTEMS MASTERY │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ TRACK 1: CORE FOUNDATIONS TRACK 2: CONSENSUS & COORDINATION │ │ ───────────────────────── ────────────────────────────── │ │ □ Why Distributed? □ The Consensus Problem │ │ □ Network Fundamentals □ Paxos (Single & Multi) │ │ □ Time & Ordering □ Raft (In-Depth) │ │ □ Consistency Models (NEW) □ Byzantine Fault Tolerance (NEW) │ │ □ Distributed Snapshots (NEW) □ Gossip Protocols (NEW) │ │ □ CAP/PACELC Theorems □ Formal Verification (TLA+) (NEW) │ │ □ Leader Election & Locks │ │ │ │ TRACK 3: REPLICATION TRACK 4: TRANSACTIONS │ │ ───────────────────── ────────────────────── │ │ □ Single-Leader □ ACID in Distributed World │ │ □ Multi-Leader □ 2PC and 3PC │ │ □ Leaderless □ Saga Pattern │ │ □ Conflict Resolution □ TCC Pattern │ │ □ CRDTs □ Distributed Locking │ │ │ │ TRACK 5: DATA SYSTEMS TRACK 6: MESSAGING & EVENTS │ │ ───────────────────── ────────────────────────── │ │ □ Partitioning Strategies (NEW) □ Kafka Deep Dive (NEW) │ │ □ Consistent Hashing □ Event Sourcing & CQRS │ │ □ Distributed Databases □ Message Queue Patterns │ │ □ Distributed Storage □ Exactly-Once Semantics │ │ □ Stream Processing □ Dead Letter Queues │ │ │ │ ═══════════════════════════════════════════════════════════════════════════│ │ ADVANCED PRODUCTION TRACKS │ │ ═══════════════════════════════════════════════════════════════════════════│ │ │ │ TRACK 7: CLOCK SYNCHRONIZATION TRACK 8: FAULT TOLERANCE │ │ ─────────────────────────────── ──────────────────────── │ │ □ TrueTime Deep Dive □ Circuit Breaker Patterns │ │ □ GPS & Atomic Clocks □ Bulkhead Isolation │ │ □ Hybrid Logical Clocks □ Retry Strategies │ │ □ Clock Synchronization □ Timeout Patterns │ │ □ Spanner's Architecture □ Graceful Degradation │ │ │ │ TRACK 9: DISTRIBUTED CACHING TRACK 10: PRODUCTION & PRACTICE │ │ ───────────────────────────── ───────────────────────────── │ │ □ Cache Strategies □ Observability at Scale │ │ □ Cache Invalidation □ Chaos Engineering │ │ □ Redis/Memcached Patterns □ Real-World Case Studies │ │ □ CDN Caching □ Interview Practice Problems │ │ □ Cache Stampede Prevention □ Staff+ Level Problem Sets │ │ │ │ CAPSTONE PROJECTS (Module 51) │ │ ──────────────────────────── │ │ □ Build Distributed KV Store │ │ □ Implement Raft Consensus │ │ □ Design Interview Practice │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` *** ## Track 1: Foundations Build the mental models that all distributed systems are built upon. **Duration**: 4-6 hours Understanding the fundamental reasons and challenges. * Single machine limitations (CPU, memory, disk, network) * Horizontal vs Vertical scaling trade-offs * The Eight Fallacies of Distributed Computing (deep dive) * Types: Compute clusters, Storage systems, Coordination systems * Real examples: Google's evolution from single machines to global infrastructure **Interview Focus**: Why not just use a bigger machine? Cost-benefit analysis **Duration**: 6-8 hours Networking knowledge every distributed systems engineer needs. * TCP guarantees and failure modes * Network partitions: What they are and how to detect them * Message passing: At-most-once, at-least-once, exactly-once * RPC frameworks: gRPC, Thrift, Protocol Buffers * Failure detection: Heartbeats, Phi accrual detector **Interview Focus**: How do you detect if a node is dead vs slow? **Duration**: 8-10 hours **Critical Topic**: Time is the foundation of distributed systems reasoning. * Why physical clocks fail (NTP drift, leap seconds) * Logical clocks (Lamport timestamps) * Vector clocks (causality tracking) * Hybrid logical clocks (HLC) * TrueTime (Google's GPS + atomic clock approach) * Happens-before relationship **Interview Focus**: How does Google Spanner achieve external consistency? **Duration**: 4-6 hours Understanding what can go wrong is crucial for designing resilient systems. * Fail-stop vs Fail-recover * Byzantine failures * Network failures: Partition, delay, reordering * Partial failures: The hardest problem * Gray failures (subtle, hard-to-detect issues) **Interview Focus**: Design for partial failures in a payment system **Duration**: 6-8 hours The fundamental trade-offs in distributed systems. * CAP Theorem: Proof and implications * CP vs AP: Real-world examples * PACELC: The more practical framework * Beyond CAP: Harvest and Yield * Consistency spectrum: Linearizable → Eventual **Interview Focus**: Is your system CP or AP? What are the trade-offs? **Duration**: 6-8 hours Capturing a consistent global state in a distributed system. * The Global State Problem: Why we can't just "freeze" time * Chandy-Lamport Algorithm: Markers and state recording * Consistent vs Inconsistent cuts * Practical uses: Distributed debugging, Checkpointing, Termination detection **Interview Focus**: How do you take a backup of a distributed database without stopping writes? **Duration**: 6-8 hours Decentralized communication and membership. * Epidemic algorithms: Rumor mongering and anti-entropy * SWIM: Scalable Weakly-consistent Infection-style Process Group Membership * Phi Accrual Failure Detection: Suspicion-based detectors * Use cases: Cassandra membership, Redis Cluster, HashiCorp Serf/Consul **Interview Focus**: How does a 1000-node cluster detect a single node failure without a central master? *** ## Track 2: Consensus Protocols The heart of distributed systems. Master these for Staff+ interviews. **Duration**: 4-6 hours Why consensus is hard and why it matters. * FLP Impossibility result (and its implications) * Safety vs Liveness guarantees * Consensus use cases: Leader election, configuration, transactions * Relationship to State Machine Replication **Interview Focus**: Can you achieve consensus in an asynchronous system? **Duration**: 10-12 hours **The Original**: Understanding Paxos is fundamental. * Basic Paxos: Prepare/Promise, Accept/Accepted * Why Paxos works (safety proofs) * Multi-Paxos optimizations * Paxos Made Simple (Lamport's paper walkthrough) * Fast Paxos and Flexible Paxos * EPaxos (Leaderless variant) **Interview Focus**: Walk through a Paxos round with failures **Duration**: 12-14 hours **Most Asked**: Raft is the go-to consensus protocol in interviews. * Leader election mechanism * Log replication and commit rules * Safety properties and proofs * Membership changes (joint consensus) * Log compaction and snapshots * Raft vs Paxos comparison * etcd/Consul implementation details **Hands-On**: Implement Raft from scratch **Interview Focus**: What happens when the leader fails during commit? **Duration**: 4-6 hours An alternative view of consensus. * VR protocol overview * View changes and recovery * Comparison with Raft * When to use VR vs Raft **Duration**: 6-8 hours How Zookeeper maintains distributed coordination. * ZAB protocol phases * Leader activation and synchronization * Recovery and failover * Zookeeper guarantees (FIFO, linearizable writes) * Zookeeper use cases: Locking, configuration, leader election **Interview Focus**: Design a distributed lock using Zookeeper **Duration**: 8-10 hours Handling malicious nodes and arbitrary failures. * The Byzantine Generals Problem: Understanding the consensus bound (3f+1) * PBFT: Practical Byzantine Fault Tolerance internals * Modern BFT: Tendermint, HotStuff (used in Libra/Diem) * Proof of Work vs BFT: Performance and safety trade-offs **Interview Focus**: How do you achieve consensus when some nodes might be actively lying or malicious? **Duration**: 8-10 hours Proving correctness before writing code. * Why testing isn't enough: State space explosion * TLA+ and PlusCal basics * Modeling safety (invariants) vs liveness (progress) * The TLC Model Checker and error traces * Case studies: How AWS and MongoDB use TLA+ **Interview Focus**: How do you prove that your custom consensus protocol is correct? *** ## Track 3: Replication Strategies How data is copied and kept consistent across nodes. **Duration**: 6-8 hours The simplest and most common replication strategy. * Synchronous vs Asynchronous replication * Semi-synchronous replication * Replication lag and its problems * Read-your-writes, Monotonic reads, Consistent prefix * Failover handling and split-brain prevention * MySQL/PostgreSQL replication internals **Interview Focus**: How do you handle replication lag in a user-facing feature? **Duration**: 8-10 hours When single-leader isn't enough. * Use cases: Multi-datacenter, offline clients * Conflict detection and resolution * Last-write-wins (LWW) and its problems * Custom conflict resolution logic * CockroachDB and TiDB approach **Interview Focus**: Design multi-region writes for a collaborative editor **Duration**: 8-10 hours The Dynamo-style approach used by Cassandra, Riak, etc. * Read/write quorums (R + W > N) * Sloppy quorums and hinted handoff * Anti-entropy: Read repair, Merkle trees * Dynamo paper deep dive * Cassandra consistency levels **Interview Focus**: When would you choose leaderless over leader-based? **Duration**: 6-8 hours When conflicts happen, how do you resolve them? * Application-level resolution * Version vectors * LWW strategies and pitfalls * Merge functions * Operational transformation (Google Docs) **Interview Focus**: Design conflict resolution for a shopping cart **Duration**: 8-10 hours **Advanced Topic**: Conflict-free Replicated Data Types. * Operation-based vs State-based CRDTs * G-Counter, PN-Counter * G-Set, 2P-Set, OR-Set * LWW-Register, MV-Register * CRDT-based databases (Riak, Redis CRDT) * Performance and memory implications **Interview Focus**: Design a collaborative text editor using CRDTs *** ## Track 4: Distributed Transactions Maintaining data integrity across multiple nodes. **Duration**: 6-8 hours How ACID properties translate to distributed environments. * Local vs Distributed transactions * Isolation levels: Read uncommitted → Serializable * Snapshot isolation and write skew * Serializable Snapshot Isolation (SSI) **Interview Focus**: What isolation level would you choose for a banking system? **Duration**: 8-10 hours The classic distributed transaction protocol. * Prepare and Commit phases * Coordinator failures and blocking * Participant failures and recovery * 2PC in practice: XA transactions * Why 2PC is often avoided (performance, availability) **Interview Focus**: What happens if the coordinator crashes after prepare? **Duration**: 4-6 hours Attempting to solve 2PC's blocking problem. * Pre-commit phase addition * Non-blocking under certain failures * Why 3PC isn't commonly used * Network partition problems **Duration**: 8-10 hours **Production Favorite**: Long-running transactions without locks. * Choreography vs Orchestration * Compensating transactions * Semantic locks and countermeasures * Saga execution coordinator * Saga pattern in microservices * Temporal.io and Cadence workflows **Hands-On**: Implement an order saga with compensation **Interview Focus**: Design a travel booking saga with compensation logic **Duration**: 4-6 hours Try-Confirm-Cancel for distributed transactions. * Two-phase approach at application level * Resource reservation * Timeout handling * When to use TCC vs Saga **Interview Focus**: TCC vs 2PC vs Saga - when to use each? **Duration**: 6-8 hours Coordinating access to shared resources. * Single-node locks in distributed systems (Redis SETNX) * Redlock algorithm and its critique * Fencing tokens for safety * Zookeeper-based locks * Lease-based locking **Interview Focus**: Design a distributed rate limiter with locks *** ## Track 5: Data Systems at Scale Partitioning, storage, and processing at massive scale. **Duration**: 8-10 hours How to split data across nodes effectively. * Key-range partitioning * Hash partitioning * Hybrid approaches * Secondary indexes: Local vs Global * Rebalancing strategies * Hot spots and skew handling **Interview Focus**: How would you partition a social network's posts? **Duration**: 6-8 hours The foundational algorithm for distributed systems. * Basic consistent hashing * Virtual nodes for load balancing * Bounded-load consistent hashing * Jump consistent hashing * Rendezvous hashing (HRW) **Hands-On**: Implement consistent hashing with virtual nodes **Interview Focus**: Design a distributed cache with consistent hashing **Duration**: 12-14 hours How production databases work internally. * **Spanner**: TrueTime, external consistency, Paxos groups * **CockroachDB**: Raft, serializable isolation, SQL distribution * **TiDB**: Raft + Percolator, hybrid OLTP/OLAP * **Cassandra**: Gossip, consistent hashing, tunable consistency * **DynamoDB**: Leaderless, GSI, adaptive capacity * **MongoDB**: Raft-based replication, sharding **Interview Focus**: How does Spanner achieve global consistency? **Duration**: 8-10 hours Block and object storage at scale. * GFS/HDFS architecture * Object storage (S3 architecture) * Erasure coding for durability * Ceph architecture * Tiered storage strategies **Interview Focus**: Design a petabyte-scale storage system **Duration**: 10-12 hours Real-time data processing at scale. * Event sourcing and event-driven architecture * Kafka internals: Partitions, consumer groups, exactly-once * Stream processing: Flink, Kafka Streams * Windowing: Tumbling, Sliding, Session * Watermarks and late data handling * Exactly-once semantics in streaming **Interview Focus**: Design a real-time analytics pipeline *** ## Track 6: Production Excellence Operating distributed systems at scale. **Duration**: 6-8 hours You can't fix what you can't see. * Distributed tracing (Jaeger, Zipkin, OpenTelemetry) * Metrics aggregation at scale * Log aggregation and analysis * Correlation across services * SLIs, SLOs, and error budgets **Interview Focus**: How do you debug a latency spike across 100 services? **Duration**: 6-8 hours Netflix-style reliability through controlled chaos. * Chaos Monkey and the Simian Army * Designing chaos experiments * Blast radius control * Failure injection frameworks (Litmus, Chaos Mesh) * Game days and runbooks **Interview Focus**: How would you test your system's resilience? **Duration**: 6-8 hours Keeping systems running at scale. * Toil reduction and automation * On-call best practices * Postmortem culture (blameless) * Error budgets and release velocity * Progressive rollouts **Interview Focus**: Describe your approach to a 50% latency increase **Duration**: 6-8 hours Designing systems that are inherently resistant to failure. * Static Stability and over-provisioning * Cell-based Architectures and blast radius control * Dependency Isolation (The Bulkhead Pattern) * Avoiding control-plane dependencies in recovery paths **Interview Focus**: How do you design a system that survives an AZ failure without autoscaling? **Duration**: 4-6 hours When things go wrong at scale. * Incident response playbooks * Communication during outages * Escalation procedures * Root cause analysis * Learning from failures **Duration**: 6-8 hours Ensuring your system can handle growth. * Load testing strategies * Capacity modeling * Performance regression detection * Autoscaling strategies * Cost optimization at scale **Interview Focus**: How do you prepare for a 10x traffic spike? *** ## Track 7: Clock Synchronization (Advanced) Time is the foundation of distributed systems. Master clock synchronization for Staff+ expertise. **Duration**: 6-8 hours How clocks stay synchronized across networks. * NTP architecture and stratum levels * PTP/IEEE 1588 for microsecond precision * Clock drift detection and correction * Network asymmetry compensation * Monitoring clock health in production **Interview Focus**: How do you detect and handle clock skew in your system? **Duration**: 8-10 hours Capturing causality without physical time. * Lamport timestamps and happened-before relationship * Vector clocks for precise conflict detection * Comparison rules and concurrency proofs * Implementation in DynamoDB and Riak **Interview Focus**: When would you use vector clocks vs logical clocks? **Duration**: 8-10 hours Combining physical and logical time for practical systems. * HLC design and implementation * Timestamp encoding strategies * CockroachDB's MVCC with HLC * Causality tracking with bounded skew * HLC vs Vector Clocks trade-offs **Hands-On**: Implement HLC for a distributed database **Interview Focus**: Why choose HLC over pure logical clocks? **Duration**: 10-12 hours **Critical Topic**: Google Spanner's revolutionary approach to time. * GPS time transfer and accuracy bounds * Atomic clock drift characteristics (Rubidium vs Cesium) * TrueTime API: `TT.now()`, `TT.after()`, `TT.before()` * Uncertainty intervals and commit-wait protocol * How Spanner uses TrueTime for external consistency **Interview Focus**: Walk through how Spanner commits a transaction using TrueTime *** ## Track 8: Fault Tolerance Patterns Building resilient systems that survive failures. **Duration**: 8-10 hours **Production Essential**: Prevent cascade failures. * State machine: Closed → Open → Half-Open * Failure threshold configuration * Timeout and retry integration * Hystrix and Resilience4j implementations * Monitoring circuit breaker health **Hands-On**: Implement a circuit breaker with state transitions **Interview Focus**: Design circuit breakers for a payment gateway **Duration**: 6-8 hours Contain failures to prevent system-wide outages. * Thread pool isolation patterns * Semaphore-based bulkheads * Connection pool partitioning * Process-level isolation * Kubernetes resource limits as bulkheads **Interview Focus**: How do you prevent one slow service from affecting others? **Duration**: 6-8 hours When and how to retry failed operations. * Exponential backoff with jitter * Retry budgets and thundering herd prevention * Idempotency keys for safe retries * Deadline propagation across services * Distinguishing transient vs permanent failures **Interview Focus**: Design a retry strategy for a distributed task queue **Duration**: 8-10 hours Maintaining partial functionality during failures. * Feature flags for degradation * Fallback strategies and stale data serving * Load shedding and admission control * Quality-of-service tiering * Netflix's fallback hierarchies **Interview Focus**: How would you degrade an e-commerce site during database issues? **Duration**: 4-6 hours The most important but often misunderstood pattern. * Connection vs read vs write timeouts * Timeout cascades and deadline propagation * Context cancellation across service boundaries * Calculating appropriate timeout values * Timeout vs circuit breaker interaction **Interview Focus**: How do you set timeouts for a microservices call chain? *** ## Track 9: Distributed Caching Caching patterns for high-performance distributed systems. **Duration**: 8-10 hours Choosing the right caching pattern for your use case. * Cache-aside (lazy loading) pattern * Read-through and write-through caching * Write-behind (write-back) caching * Refresh-ahead pattern * Cache eviction policies (LRU, LFU, TTL) **Interview Focus**: When would you choose write-behind over write-through? **Duration**: 8-10 hours **The Hard Problem**: Keeping caches consistent. * TTL-based invalidation strategies * Event-driven invalidation with Kafka/pub-sub * Tag-based cache invalidation * Cascading invalidation patterns * Cache versioning strategies **Interview Focus**: Design cache invalidation for a product catalog **Duration**: 10-12 hours Production-grade distributed cache implementations. * Redis Cluster architecture and slot migration * Redis Sentinel for high availability * Memcached consistent hashing * Memory management and eviction * Replication lag and read consistency * Redis vs Memcached decision framework **Interview Focus**: Design a distributed session store using Redis **Duration**: 6-8 hours Caching at the edge for global performance. * CDN architecture and PoP design * Cache-Control header strategies * Origin shield and tiered caching * Cache purging at scale * Dynamic content caching patterns **Interview Focus**: Design CDN caching for a video streaming platform **Duration**: 6-8 hours Preventing thundering herd on cache misses. * Locking and mutex patterns * Probabilistic early expiration * Request coalescing * Background refresh strategies * Circuit breaker integration **Interview Focus**: How do you handle cache stampede during Black Friday? *** ## Special Track: Real-World Case Studies Learn from production systems at scale. **Duration**: 8-10 hours The first globally distributed, strongly consistent database. * TrueTime and external consistency * Paxos groups and data placement * Lock-free read-only transactions * Schema changes without downtime * Real failure stories and lessons learned **Interview Focus**: How does Spanner achieve 5 nines availability? **Duration**: 8-10 hours The paper that launched NoSQL. * Consistent hashing with virtual nodes * Vector clocks and conflict resolution * Sloppy quorums and hinted handoff * Evolution from Dynamo to DynamoDB * Global Tables and cross-region replication **Interview Focus**: Design a shopping cart using Dynamo-style storage **Duration**: 6-8 hours Chaos engineering pioneers. * Chaos Monkey and Simian Army * EVCache and caching at scale * Zuul gateway and load shedding * Failure injection testing * Multi-region active-active deployment **Interview Focus**: Design a chaos engineering strategy for your system **Duration**: 6-8 hours Building reliable systems for millions of rides. * Ringpop for membership and routing * Cadence/Temporal workflow orchestration * Geospatial indexing at scale * Real-time dispatch and matching * Multi-region failover strategies **Interview Focus**: Design a ride-matching system like Uber *** ## Staff+ Interview Practice Problems Curated problems for senior-level interviews. Design a rate limiter that works across multiple data centers with sub-millisecond overhead Build a transaction coordinator supporting 2PC, Saga, and TCC patterns Design a leaderboard supporting millions of concurrent players with real-time updates Design a database with strong consistency guarantees across continents **Interview Tip**: Each problem includes detailed solutions, trade-off analysis, and follow-up questions commonly asked at Google, Meta, Amazon, and other top companies. *** ## Capstone Projects Apply everything you've learned. Build a key-value store with: * Raft-based replication * Consistent hashing for partitioning * Read/write quorums * Snapshot and recovery A complete Raft implementation: * Leader election * Log replication * Membership changes * Persistence and recovery Build a coordination service: * Ephemeral nodes * Watch mechanism * Sequential ordering * Lock implementation Practice system design: * Design Uber's dispatch system * Design Stripe's payment processing * Design Netflix's CDN * Design Twitter's timeline *** ## Key Papers to Read Essential reading for deep understanding: | Paper | Why It Matters | | ------------------------------- | ----------------------------------------------------------- | | **Dynamo** (Amazon) | Leaderless replication, vector clocks, eventual consistency | | **Spanner** (Google) | TrueTime, globally consistent transactions | | **Raft** (Stanford) | Understandable consensus | | **Paxos Made Simple** (Lamport) | The original consensus paper | | **MapReduce** (Google) | Distributed computation paradigm | | **Kafka** (LinkedIn) | Distributed log architecture | | **Time, Clocks** (Lamport) | Logical time foundations | | **CALM Theorem** | Consistency without coordination | | **FLP Impossibility** | Limits of distributed consensus | | **Harvest/Yield** | Practical CAP trade-offs | *** ## Interview Preparation Strategy Complete Tracks 1-3. Focus on: * CAP/PACELC intuition * Raft protocol (draw from memory) * Replication trade-offs Complete Track 4. Be ready to: * Design a saga for any use case * Explain 2PC failure modes * Discuss distributed locking trade-offs Complete Track 5. Know: * How Spanner achieves global consistency * How Kafka provides exactly-once * When to use Cassandra vs PostgreSQL Complete Tracks 7-9 (New!). Focus on: * TrueTime and clock synchronization * Circuit breakers and fault tolerance * Distributed caching patterns Review real-world architectures: * Google Spanner, Amazon Dynamo * Netflix resilience, Uber real-time * Learn from actual failure post-mortems * Practice 3-4 system design problems per week * Use the Staff+ Interview Problems module * Focus on distributed aspects * Record yourself and review *** ## Who This Course Is For * **Senior Engineers** (4+ years) aiming for Staff/Principal * **Backend Engineers** wanting deep distributed systems knowledge * **Infrastructure Engineers** building platforms * **Anyone targeting FAANG/top-tier companies** * Strong programming in at least one language * Basic networking (TCP/IP, HTTP) * Database fundamentals (SQL, transactions) * Some system design exposure * **Full-time study**: 10-14 weeks * **Part-time (10-15 hrs/week)**: 18-24 weeks * **Each module**: 4-14 hours * **Projects**: 20-40 hours each * **New tracks (7-9)**: Additional 6-8 weeks *** ## Ready to Begin? Begin with Track 1 to build your mental model of distributed systems Learn from Google, Amazon, Netflix, and Uber's production systems Staff+ level problems with detailed solutions and trade-off analysis Circuit breakers, retries, and graceful degradation patterns *** ## Interview Deep-Dive **Strong Answer:** * The first thing I clarify is what the system's core invariants are. For a payment ledger, an incorrect balance is catastrophic, so I lean CP. For a social media feed, a user seeing a slightly stale timeline for a few seconds is acceptable, so AP makes sense. * I never frame it as a binary choice for the entire system. Most real systems are a mix: the metadata service might be CP (using Raft-backed etcd), while the content-delivery layer is AP (eventual consistency with CDN caching). * I also move beyond CAP to PACELC, because CAP only describes behavior during partitions. During normal operation, the trade-off is between latency and consistency. A system like Spanner is PC/EC -- it chooses consistency even when there is no partition, accepting higher latency for correctness. DynamoDB is PA/EL -- it prioritizes availability during partitions and low latency during normal operation. * In practice, I would enumerate the critical user-facing flows, classify each by its tolerance for staleness or unavailability, and then assign the appropriate consistency level per flow rather than per system. **Follow-up: How would you handle the situation where a single user request touches both a CP subsystem and an AP subsystem?** At the boundary between CP and AP subsystems, you need to be very deliberate about what guarantees the end user actually sees. The pattern I use is to make the CP subsystem the "source of truth" for correctness-critical state (like account balance or inventory count) and let the AP subsystem handle best-effort, eventually-consistent derived data (like recommendation scores or timeline caches). The user-facing request first commits to the CP path -- for example, debiting inventory via a Raft-backed service -- and then asynchronously propagates to the AP path via an event bus like Kafka. If the AP path is temporarily inconsistent, the UI can show a "processing" state or fall back to the last-known-good value. The key principle is: never let the AP subsystem's staleness corrupt the CP subsystem's invariants. The CP path is the gatekeeper; the AP path is the fast lane. **Strong Answer:** * I would say: "CAP means that when our network has a serious problem -- a partition -- we have to pick between two options. Option one: we stop serving some requests to make sure every answer is correct (consistency). Option two: we keep serving every request, but some answers might be outdated (availability). We cannot have both during the outage." * The warning I would give is that CAP is about the extreme case -- a network partition. Most of the time, partitions are rare and brief. The real day-to-day trade-off is between latency and consistency (PACELC). A VP should care more about "how fast are we during normal operation?" than "what happens during an outage that occurs once a year?" * I would also warn against the common misconception that "AP means our data is unreliable." AP systems like DynamoDB or Cassandra are used by Amazon and Netflix for mission-critical workloads. They are not unreliable -- they just define consistency differently and resolve conflicts after the fact. **Follow-up: A senior engineer on your team argues that since partitions are rare, you should always choose CP. How do you respond?** I would push back respectfully. Partitions are rare, but their effects are amplified. During a partition, a CP system rejects writes, which means user-facing errors or degraded functionality for the duration. If your SLA requires 99.99% availability, even a 5-minute partition per month can blow your error budget. The right answer depends on the business impact: is a wrong answer worse than no answer? For a bank ledger, yes -- CP is correct. For a shopping cart, no -- Amazon's original Dynamo paper showed that losing a cart addition costs more revenue than showing a slightly stale cart. I would bring data: what is the expected frequency of partitions, what is the business cost of unavailability vs. inconsistency, and use that to drive the decision rather than an abstract preference. **Strong Answer:** * In almost every greenfield project today, I would default to Raft. The reason is understandability. Raft was explicitly designed to be easier to implement correctly, and correctness in consensus is everything. Bugs in consensus lead to data loss, split-brain, or silent corruption -- the worst kind of production incidents. * I would choose Paxos (specifically Multi-Paxos or a variant like EPaxos) in specific scenarios: when I need flexible quorum configurations, when I need leaderless consensus for lower latency in geo-distributed deployments, or when I am extending an existing system that already uses Paxos (like Google's internal infrastructure). * The practical trade-off: Raft has a single leader, which simplifies reasoning but creates a throughput bottleneck and a latency penalty for writes that must cross regions. EPaxos removes the leader but is significantly harder to implement and reason about. * In production, I look at the ecosystem too. etcd (Raft) and Consul (Raft) have mature, battle-tested implementations. If I am building a coordination service, I would use one of these rather than implementing consensus from scratch. **Follow-up: What happens in a Raft cluster if the leader is in a data center that becomes partitioned from the majority?** The partitioned leader will stop receiving acknowledgments from the majority and its heartbeats will not reach most followers. The followers in the majority partition will time out, increment their term, and elect a new leader. The old leader, now in the minority partition, will eventually receive a message with a higher term number and step down to follower. During the transition, there is a brief period where no leader exists and writes are rejected -- this is the availability cost of CP consensus. The critical safety property is that the old leader cannot commit any new entries because it cannot reach a majority. Any entries it accepted but did not commit will be overwritten by the new leader's log. This is by design: Raft sacrifices availability during partitions to guarantee that committed entries are never lost. **Strong Answer:** * A well-known example is the early days of Amazon's shopping cart. They chose strong consistency initially, which meant that during network hiccups between data centers, cart operations would fail or time out. Users would click "Add to Cart" and see an error. Amazon calculated that every failed cart addition had a direct revenue cost, so they moved to an eventually consistent model (Dynamo) where the cart would always accept writes, even if it meant occasionally showing duplicate items or slightly stale state. The trade-off was worth it: a cart with an extra item is a minor UI annoyance, but a cart that refuses to work loses a sale. * Another example is the MongoDB Jepsen findings. Early versions of MongoDB claimed "strong consistency" but under network partitions, stale reads were possible because reads could be served by secondaries that had not replicated the latest write. This was not a bug in the consistency model itself but a mismatch between what was advertised and what was actually implemented -- a common and dangerous failure mode. * The lesson for interviews: always verify that your system's actual behavior matches its claimed consistency model. Use tools like Jepsen, write linearizability checkers, and test under real failure conditions (network partitions, clock skew, process crashes). **Follow-up: How would you set up ongoing validation that your system actually provides the consistency guarantees it claims?** I would implement three layers of validation. First, a continuous integration Jepsen-style test suite that runs against a staging cluster, injecting network partitions, clock skew, and process kills while verifying linearizability or whatever consistency model you claim. Second, production-level consistency auditing: for critical paths like financial transactions, I would dual-write to an audit log and run an offline checker that reconstructs the serial order and flags any anomalies. Third, observability: track metrics like replication lag, leader election frequency, and quorum failures. If replication lag spikes above your SLA threshold, that is an early warning that your consistency guarantees are at risk before users notice. The goal is to detect consistency violations before they become customer-visible incidents.