​

Part XXXIV — Capacity Planning and Back-of-Envelope Estimation

​

Chapter 41: Capacity Planning

​

41.1 Numbers Every Engineer Should Know

These latency numbers (originally from Jeff Dean, updated for modern hardware) are essential for back-of-envelope calculations and system design interviews: Useful throughput numbers:

• A single PostgreSQL instance: ~5,000-20,000 simple queries/second
• Redis: ~100,000 operations/second
• A single web server (Node.js/Go): ~10,000-50,000 requests/second for simple endpoints
• Kafka: ~1 million messages/second per broker (small messages)
• 1 Gbps network: ~125 MB/second theoretical throughput

Useful size and time numbers:

---

​

Back-of-Envelope Cheat Sheet

Use this reference table during system design interviews and architecture sessions. These are approximate values — the point is speed, not decimal precision. Storage Sizes Request Rate Conversions Typical Latencies

---

​

41.2 Back-of-Envelope Estimation

The skill of quickly estimating whether a design will work at the required scale. This comes up constantly in system design interviews and real architecture discussions. The process: Start with the requirement (e.g., “store 5 years of user activity”). Estimate the data size per event. Estimate the volume (events/day x 365 x 5). Multiply. Add overhead (indexes, replication, headroom). Compare against known limits of the technology you are considering. Example — URL Shortener storage: 100 million new URLs/month. Each record: short code (7 bytes) + original URL (average 200 bytes) + metadata (50 bytes) = ~257 bytes. Per month: 100M x 257 bytes = ~25.7 GB. Per year: ~308 GB. Over 5 years: ~1.5 TB. This fits easily in a single PostgreSQL instance. No need for sharding — the problem is read throughput (caching), not storage. Example — Chat message throughput: 10 million daily active users, average 50 messages/day = 500 million messages/day = ~5,787 messages/second average, ~17,000 messages/second peak (3x). Each message: ~500 bytes. Daily storage: 500M x 500 bytes = 250 GB/day. This helps decide: do we need Kafka? (Yes, for the throughput.) Do we need sharding? (Eventually, for storage.) Can a single database handle it? (Not long-term.)

---

​

Worked Example: Design Twitter — Estimate Storage for 500M Users, 200M Daily Active

This is a complete back-of-envelope walkthrough combining multiple estimation skills. Step 1: Define the scope.

• 500M total users, 200M daily active users (DAU)
• Features to estimate: user profiles, tweets, timelines, media, follows graph

Step 2: User Profiles.

• 500M users x ~2 KB per profile (name, bio, email, settings, timestamps) = 1 TB
• With 3x replication = ~3 TB
• This is trivially handled by a single sharded database cluster

Step 3: Tweets.

• Average DAU posts 2 tweets/day: 200M x 2 = 400M tweets/day
• Each tweet: 280 bytes (text) + 200 bytes (metadata) = ~480 bytes
• Daily tweet storage: 400M x 480 = 192 GB/day
• Yearly: 192 x 365 = ~70 TB/year
• 5 years: ~350 TB of raw tweet data
• With replication (3x) + indexes (~30%): ~350 x 3 x 1.3 = ~1.4 PB

Step 4: Media (images, videos).

• Assume 20% of tweets have an image (~200 KB avg compressed) and 5% have a video (~5 MB avg compressed)
• Images: 400M x 0.20 x 200 KB = 16 TB/day
• Videos: 400M x 0.05 x 5 MB = 100 TB/day
• Media dominates storage. Per year: ~42 PB. This goes in object storage (S3), not a database.

Step 5: Timeline reads (throughput).

• 200M DAU, each opens the app 5 times/day, each load fetches ~50 tweets
• Timeline reads: 200M x 5 = 1B timeline loads/day = ~11,600 requests/second (avg), ~35,000 peak
• Each timeline load fans out to fetch tweets from followed users
• This demands heavy caching (pre-computed timelines in Redis) and a fan-out architecture

Step 6: Follow graph.

• Average user follows 200 accounts: 500M x 200 = 100B edges
• Each edge: 8 bytes (follower_id) + 8 bytes (followee_id) + 8 bytes (timestamp) = ~24 bytes
• Total: 100B x 24 = ~2.4 TB (manageable in a graph database or sharded relational DB)

Step 7: Summary.

---

​

Real-World Story: How Google Manages Billions of Lines of Code in a Single Repository

Google stores virtually all of its code — billions of lines across tens of thousands of projects — in a single monolithic repository. As of their 2016 paper, the repository contained over 2 billion lines of code across 9 million source files, with 86 TB of content and roughly 45,000 commits per day from 25,000+ developers. They do not use Git for this. Git was not designed for a repository of this scale — operations like git status and git clone would take hours or days on a repo this size. Instead, Google built a custom version control system called Piper, backed by their distributed file system (Bigtable and Colossus). Developers do not clone the entire repo. They use a virtual filesystem called CitC (Clients in the Cloud) that presents the illusion of having the full repo locally, while fetching only the files you actually open or build. Why a monorepo? Because dependency management across thousands of projects becomes tractable. If Team A changes an API, they can also update every caller of that API across the company in a single commit (called a “large-scale change” or LSC). This eliminates dependency versioning hell — there is only one version of every library: the one at HEAD. Google built custom tooling (Rosie, TAP) to automatically test the impact of changes across the entire codebase before they land. The lesson for capacity estimation: Google’s monorepo is a masterclass in understanding your scale constraints and building tools to match. They estimated that a standard VCS could not handle their volume, validated that estimate, and built custom infrastructure. They did not guess — they did the math. 86 TB of code. 45,000 commits/day. 800,000 queries/second to the version control backend. Those numbers told them exactly what architecture they needed.

---

​

Part XXXV — Git Workflows and Code Review

​

Chapter 42: Git Workflows

​

42.1 Branching Strategies

Trunk-Based Development: All developers commit to main (or trunk). Short-lived feature branches (hours, not weeks). Feature flags hide incomplete work. CI runs on every commit. This is the strategy used by Google, Facebook, and most high-performing teams measured in the Accelerate research. Requires strong CI, feature flags, and automated testing. GitHub Flow: Create a branch from main, do work, open a pull request, review, merge to main, deploy. Simple, effective for most teams. Branches should be short-lived (1-3 days). Long-lived branches create merge conflicts and delay integration. GitFlow: Separate develop, release, and hotfix branches alongside main. Popular but complex. Appropriate for software with formal release cycles (mobile apps, packaged software). Overkill for web services that deploy continuously. The rule: Start with trunk-based development or GitHub Flow. Only adopt GitFlow if you have a genuine need for parallel release management.

---

​

Git Workflow Comparison

​

42.2 Merge Strategies

Merge commit: Preserves full branch history. Creates a merge commit. Good for seeing exactly what a PR contained. Noisy history. Squash merge: Combines all branch commits into a single commit on main. Clean history. Loses granular commit history (which rarely matters). Most teams prefer this for PRs. Rebase: Replays branch commits on top of main. Linear history. Can cause confusion with force pushes. Best for personal workflow, risky for shared branches.

​

42.3 Code Review Best Practices

For authors: Keep PRs small (under 400 lines changed). Write a clear description (what, why, how, testing). Self-review before requesting review. Respond to feedback promptly. Do not take feedback personally. For reviewers: Review within 24 hours (review latency kills velocity). Focus on correctness, clarity, and maintainability — not style preferences (use linters for that). Ask questions instead of making demands (“What happens if this is null?” vs “This will crash on null”). Approve when it is “good enough” — do not block on perfection. Distinguish blocking issues from nits. What to look for: Logic errors. Missing edge cases. Security vulnerabilities. Performance concerns at scale. Missing tests for changed behavior. API contract changes. Error handling gaps. Race conditions.

---

​

Code Review: What to Look For (Checklist)

Use this checklist when reviewing pull requests. Not every item applies to every PR, but scanning through this list ensures you do not miss critical issues.

---

​

Code Review Anti-Patterns

Recognizing these patterns helps teams improve their review culture.

---

​

Real-World Story: How GitHub Scales Code Review Across Thousands of Engineers

GitHub itself — the company that hosts code review for millions of teams — has a distinctive code review culture worth studying. With over 3,000 employees and hundreds of engineers shipping to a monolithic Ruby on Rails application (and increasingly a set of microservices), they have had to solve the “code review at scale” problem for their own codebase, not just for their customers. How they do it: GitHub uses their own pull request workflow (naturally), but the interesting part is the culture and tooling around it. They practice what they call “review-driven development” — PRs are the unit of work, and no code reaches main without at least one substantive review. They use CODEOWNERS files extensively, which automatically assign reviewers based on which files a PR touches. This prevents the bottleneck of a few senior engineers being the review gateway for everything. Review latency as a metric: GitHub treats review latency (the time between a PR being opened and receiving its first substantive review) as a key engineering health metric. Their internal target is that most PRs receive a review within a few hours. They found that when review latency creeps up, branch lifetimes grow, merge conflicts increase, and deploy frequency drops — a cascading effect that slows the entire organization. The “ship small” principle: GitHub’s engineering blog has documented their emphasis on small, incremental PRs. They encourage engineers to break work into PRs of under 400 lines when possible. Smaller PRs get reviewed faster, reviewed more thoroughly, and merge with fewer conflicts. They use feature flags extensively to allow incomplete features to be merged behind a flag. The lesson: Code review is not just a quality gate — it is a communication mechanism. GitHub’s approach shows that the biggest leverage is not in what reviewers look for (checklists help, but they are secondary) but in how fast the review loop turns. Fast reviews mean short branches mean few conflicts mean fast deploys. Slow reviews poison the entire development cycle.

---

​

Part XXXVI — Data Pipelines and Batch Processing

​

Chapter 43: Data Engineering Fundamentals

​

43.1 ETL vs ELT

ETL (Extract, Transform, Load): Data is extracted from sources, transformed in a staging area, then loaded into the destination. Traditional approach. Transformation happens before loading, so only clean data enters the warehouse. Good when the destination has limited compute (traditional data warehouses). ELT (Extract, Load, Transform): Data is extracted and loaded raw into the destination, then transformed there. Modern approach enabled by powerful cloud data warehouses (BigQuery, Snowflake, Redshift) that have cheap, scalable compute. Faster initial loading. Transforms can be iterated without re-extraction. This is the dominant pattern today.

---

​

ETL vs ELT: The Shift and When Each Applies

​

43.2 Batch vs Stream Processing

Batch: Process accumulated data at scheduled intervals (hourly, daily). Simpler, cheaper, handles large volumes efficiently. Use for: daily reports, analytics aggregation, data warehouse loading, ML model training, end-of-day reconciliation. Stream: Process data as it arrives, continuously. More complex, higher infrastructure cost, but enables real-time insights. Use for: fraud detection, live dashboards, real-time recommendations, alerting, event-driven architectures. The practical reality: Most companies need both. Stream processing for time-sensitive operations (fraud alerts within seconds), batch processing for everything else (daily reports, historical analytics). Do not default to stream processing — it is significantly more complex to build, debug, and operate.

---

​

Real-World Story: Spotify’s Journey from Batch to Streaming

Spotify’s data infrastructure evolution is one of the best-documented cases of a company navigating the batch-to-streaming transition — and learning the hard way that “just stream everything” is not the answer. The early days (batch-first). Spotify started with a Hadoop-based batch processing architecture. Every night, massive MapReduce jobs would crunch the previous day’s listening data — billions of events — to produce the analytics, recommendations, and royalty calculations that powered the business. This worked well when Spotify had tens of millions of users. Daily batch jobs ran overnight, and by morning the data team had fresh dashboards and the recommendation models had updated features. The pain point. As Spotify scaled past 100 million users and the event volume grew into the hundreds of billions per day, the batch architecture started breaking. Jobs that used to finish in 2 hours were taking 8. The “nightly batch window” was not enough — some jobs were still running when users woke up. Worse, the batch-only model meant that features like Discover Weekly and personalized playlists could only update daily. If a user listened to jazz all morning, the recommendations would not reflect that until the next day’s batch run. The hybrid architecture. Rather than ripping out batch and going fully streaming, Spotify built a hybrid architecture. They adopted Google Cloud Dataflow (based on Apache Beam) for streaming use cases where latency mattered — real-time listening activity, live A/B test metrics, and fraud detection. But they kept batch processing (migrating from Hadoop to Google BigQuery and Scio, their open-source Scala library for Apache Beam) for the heavy analytical workloads where daily granularity was perfectly fine: royalty calculations, long-term trend analysis, and ML model training. The lesson. Spotify’s story illustrates a pattern that almost every data-intensive company discovers: you end up with both batch and streaming, not one or the other. The key skill is knowing which workloads belong where. Spotify engineers have talked publicly about the mistake of over-eagerly migrating batch workloads to streaming — the added complexity of managing streaming state, handling late-arriving events, and debugging time-windowed aggregations was not worth it for workloads where “yesterday’s data by 8 AM” was a perfectly acceptable SLA.

---

​

Batch vs Stream Processing: Detailed Comparison

​

43.3 The Modern Data Stack

Extract: Fivetran, Airbyte, Stitch (managed connectors from SaaS tools and databases to the warehouse). Load: Direct loading into BigQuery, Snowflake, Redshift, Databricks. Transform: dbt (SQL-based transformations with version control, testing, and documentation — the standard tool). Orchestrate: Airflow, Dagster, Prefect (schedule and manage pipeline dependencies). Serve: Looker, Metabase, Tableau (BI and visualization), or reverse ETL back to operational systems.

---

​

Key Tools: Apache Spark, Apache Flink, and dbt

These three tools cover the majority of modern data processing needs. Knowing when to use each is a critical skill. Apache Spark Spark is a distributed data processing engine designed for large-scale batch processing and analytics. It processes data in parallel across a cluster, making it ideal for workloads that would take hours on a single machine. Apache Flink Flink is a distributed stream processing engine designed for true real-time, event-at-a-time processing. Unlike Spark Structured Streaming (which uses micro-batches), Flink processes each event individually with very low latency. dbt (data build tool) dbt is a SQL-first transformation tool that lets analytics engineers define transformations as versioned, tested, documented SQL models. It has become the standard for the “T” in ELT.

​

Real-World Story: How Airbnb’s Data Quality Crisis Led to Minerva

Airbnb’s data quality journey is a cautionary tale that became a success story — and it is one of the best-documented examples of why data quality cannot be an afterthought. The problem. By the mid-2010s, Airbnb had grown rapidly and so had their data infrastructure. Hundreds of engineers and analysts were writing SQL queries, building dashboards, and creating derived data sets. The problem: there was no single source of truth for core business metrics. Different teams had different definitions of “active listing,” “booking,” and “revenue.” One team’s revenue dashboard would show a number 15% different from another team’s — both built from the same raw data but with different filters, joins, and business logic applied. Leadership could not trust the numbers. Meetings that should have been about strategy devolved into arguments about whose dashboard was right. The investigation. Airbnb’s data team traced the root cause: metric definitions were scattered across hundreds of SQL queries, dbt models, and notebook analyses. There was no central, authoritative definition of what “revenue” meant — gross or net? Including cancellations or not? In the host’s currency or USD? Different analysts made different reasonable choices, and over time the definitions diverged silently. The solution: Minerva. Airbnb built Minerva, a centralized metrics platform that serves as the single source of truth for all business metrics. Minerva enforces that every metric has one canonical definition — written in a structured configuration language, version-controlled, reviewed, and tested. When an analyst or a dashboard needs “revenue,” they query Minerva rather than writing their own SQL. Minerva handles the joins, filters, currency conversions, and edge cases consistently every time. Key design decisions in Minerva:

• Metric definitions as code: Every metric is defined in a YAML-like configuration file, version-controlled in Git, and subject to code review. Changing a metric definition is a PR, not an ad-hoc SQL edit.
• Dimensional consistency: Minerva enforces that metrics can be sliced by a standard set of dimensions (date, country, platform, etc.) and that these dimensions are defined once and reused.
• Data quality checks built in: Every metric computation includes automated validation — row counts, null rates, bounds checking, and comparison against previous runs.
• Self-serve with guardrails: Analysts can explore and slice metrics freely, but they cannot redefine them. New metrics go through a formal definition and review process.

The outcome. After adopting Minerva, Airbnb reported that metric discrepancies dropped dramatically. The “whose numbers are right?” debates largely disappeared. Data trust increased across the organization, and the data team could spend time building new capabilities rather than debugging inconsistencies. The lesson for interviews: When discussing data quality, Airbnb’s story shows that the problem is usually not bad data in the source — it is inconsistent interpretation in the transformation layer. The fix is not more data quality checks (though those help) — it is centralizing metric definitions so that “revenue” means exactly one thing, everywhere, always.

​

43.4 Data Quality

Bad data is worse than no data — it leads to wrong decisions. Data quality checks: schema validation (are the expected columns present with correct types?), freshness checks (is the data from today or stale?), volume checks (did we receive the expected number of rows?), null checks (are critical fields populated?), uniqueness checks (no duplicate primary keys?), referential integrity (do foreign keys point to valid records?).

---

​

Data Quality: Schema Validation, Data Contracts, and Monitoring for Drift

Data quality is not a one-time check — it is a continuous practice woven into every stage of the pipeline. Schema Validation Every pipeline stage should validate that incoming data matches the expected schema before processing. This catches breaking changes early instead of letting corrupt data flow downstream into dashboards and ML models.

• At extraction: Validate that source API responses or database exports match the expected schema (column names, types, non-null constraints)
• At loading: Enforce schema-on-write or schema-on-read validation in the warehouse. Snowflake and BigQuery support schema enforcement.
• At transformation: dbt tests validate the output schema — every model should have not_null, unique, and accepted_values tests on critical columns

Data Contracts A data contract is a formal agreement between a data producer (the team that owns the source system) and a data consumer (the analytics or ML team). It specifies:

• What fields will be present and their types
• What guarantees are made about data freshness and completeness
• What the change process looks like (producer cannot rename a field without notifying consumers)
• SLAs for data delivery (e.g., “yesterday’s data available by 6 AM UTC”)

Monitoring for Drift Even with contracts, data can drift over time. Monitor for:

---

​

Curated Resources

A collection of essential references across all three topics covered in this module. These are not filler links — each one is here because it is genuinely worth your time. Capacity Planning and Estimation

• System Design Interview by Alex Xu — the most practical guide to back-of-envelope estimation. Chapters walk through real problems step by step.
• The Google SRE Book — Capacity Planning chapter — how Google approaches capacity planning in production. Freely available online.
• “Why Google Stores Billions of Lines of Code in a Single Repository” — Rachel Potvin and Josh Levenberg (2016). The definitive paper on monorepo architecture at extreme scale. Essential for understanding how capacity constraints shape tooling decisions.

Git Workflows and Code Review

• Martin Fowler and the Trunk-Based Development community — trunkbaseddevelopment.com — the comprehensive reference on trunk-based development patterns including feature flags, branch by abstraction, and release strategies.
• Conventional Commits specification — conventionalcommits.org — a lightweight, widely adopted standard for structuring commit messages. Enables automated changelogs and semantic versioning.
• Google Engineering Practices — Code Review Guide — the best freely available guide on how to conduct and receive code reviews effectively.
• Chelsea Troy — Reviewing Pull Requests — a thoughtful, deeply practical series on what makes code review effective. Covers cognitive load, empathy in review, and how to give feedback that actually helps the author improve.
• Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim — research-backed evidence that trunk-based development, short-lived branches, and fast code review cycles predict high-performing engineering teams.

Data Pipelines and Processing

• dbt documentation and the Analytics Engineering Guide — docs.getdbt.com — the official dbt documentation is excellent. Start with the “Getting Started” tutorial, then read the Analytics Engineering Guide for the philosophy behind the tool. Understanding dbt is non-negotiable for modern data engineering interviews.
• Apache Spark documentation — spark.apache.org — the official docs, plus the Databricks blog which publishes practical Spark optimization guides and architecture patterns regularly.
• Apache Flink documentation and use cases — flink.apache.org — the official Flink docs include excellent conceptual guides on stream processing fundamentals (windowing, watermarks, exactly-once semantics) that are valuable even if you never use Flink specifically.
• Netflix Technology Blog — data pipeline posts — Netflix’s data engineering team has published some of the best real-world accounts of building and operating data pipelines at massive scale. Search for posts on their data mesh adoption, Spark usage, and pipeline reliability practices.
• Fundamentals of Data Engineering by Joe Reis and Matt Housley — the most comprehensive modern introduction to the full data engineering lifecycle. Covers extraction, loading, transformation, orchestration, and data quality.
• Designing Data-Intensive Applications by Martin Kleppmann — the chapters on batch processing (Chapter 10) and stream processing (Chapter 11) are some of the best technical writing on these topics anywhere. A must-read for anyone building or operating data systems.