Skip to main content

​
Chapter 2: Architecture Overview

The Google File System architecture is elegantly simple: a single master coordinating hundreds of chunkservers, with clients communicating directly with chunkservers for data operations. This separation of control plane (metadata on the master) from data plane (actual bytes on chunkservers) is one of GFS’s most important contributions to distributed systems design. The same pattern appears today in HDFS (NameNode vs DataNode), Kubernetes (API server vs kubelets), and virtually every modern cloud storage service. This chapter explores how these components work together to create a highly scalable distributed file system.
Chapter Goals:
  • Understand the three main components: Master, Chunkservers, and Clients
  • Learn how data flows through the system
  • Grasp the separation of control and data flow
  • Appreciate the single-master design choice

​
System Components

GFS consists of three main types of components:

​
The Master

The master is the brain of GFS — a single process that manages all metadata. At first glance, a single master looks like an obvious bottleneck and single point of failure. Skeptics in 2003 criticized this choice heavily. But the GFS team recognized a crucial insight: by keeping the master entirely out of the data path and minimizing its per-operation cost, a single machine could handle metadata for a petabyte-scale cluster. The design leverages Metadata Minimization (64 bytes per chunk, all in RAM) and Direct Data Transfer (clients talk to chunkservers for data, never routing bytes through the master) to ensure it can manage petabytes of data from thousands of clients. This “simple master, smart clients” model proved so effective that it became the default architecture for an entire generation of distributed systems.
Namespace Structure: Unlike traditional file systems with directory inodes, GFS uses a lookup table mapping full pathnames to metadata. This lookup table is stored in a prefix-compressed Radix Tree (or Hash Table) for extreme RAM efficiency.The Locking Mechanism: Since there is no directory structure, operations on /a/b/c don’t require locking the parent /a or /a/b. Instead:
  1. To create /home/user/foo, the master acquires Read Locks on /home and /home/user, and a Write Lock on /home/user/foo.
  2. This allows concurrent file creations in the same directory (e.g., /home/user/bar can be created simultaneously as it only needs a read lock on the parent).
  3. No deadlock risk: Locks are acquired in a consistent lexicographical order.
Benefit: Extreme concurrency for metadata operations compared to traditional POSIX dentry locking.
What Gets Persisted:

​
Chunkservers

Chunkservers are the workhorses that store actual data:

Storage Model

How Chunks Are Stored:
  • Each chunk: 64MB max
  • Stored as Linux file on local disk
  • Named by chunk handle (globally unique)
  • File path: /gfs/chunks/<chunk_handle>
  • Checksums for each 64KB block
  • Multiple chunks per disk

Responsibilities

What Chunkservers Do:
  • Store and retrieve chunks
  • Verify data integrity (checksums)
  • Replicate chunks to other servers
  • Report to master via heartbeat
  • Delete garbage chunks
  • Handle client read/write requests

No Metadata Cache

Simple Design:
  • Chunkservers don’t cache metadata
  • Don’t track which files chunks belong to
  • Master tells them what to do
  • Simplifies consistency
  • Reduces memory requirements

Replication

Chunk Copies:
  • Each chunk replicated 3x (default)
  • Replicas on different racks
  • Replicas can serve reads
  • One primary for writes (leased)
  • Replicas forward writes in chain
Chunkserver State:

​
Clients

GFS clients are library code linked into applications:

​
Data Flow Patterns

Understanding how data flows through GFS is crucial. The key principle is the Separation of Control Flow and Data Flow. Control flows from client to master and then to the primary, while data is pushed linearly along a chain of chunkservers to maximize network throughput. This separation is one of the most practically important ideas in the paper. By decoupling “where should I write?” (control) from “here is the actual data” (data), GFS ensures the master never becomes a bandwidth bottleneck. Modern systems like Apache Kafka use the same principle: the controller handles partition assignments and leader election, but producers and consumers transfer data directly to/from broker nodes.

​
The Pipeline Push Mechanism

To fully utilize each machine’s network bandwidth, data is pushed along a chain of chunkservers rather than in a star topology.

​
Read Operation

1

Client Requests Metadata

2

Client Contacts Chunkserver

3

Data Validation

Complete Read Flow:

​
Write Operation

Writes are more complex due to maintaining consistency across replicas:

​
Record Append Operation

The record append is GFS’s “killer feature”:

​
Architecture Deep Dive

​
Chunk Size: Why 64MB?

The 64MB chunk size is a defining characteristic of GFS:
Benefits of 64MB Chunks:
  1. Reduced Metadata:
  2. Fewer Network Hops:
    • Client requests chunk locations once
    • Works with chunk for extended period
    • Reduces master load significantly
  3. Better TCP Performance:
    • Long-lived TCP connections
    • Connection setup cost amortized
    • Better throughput utilization
  4. Data Locality:
    • MapReduce can schedule entire chunk to one worker
    • Reduces network transfer
    • Better cache utilization
Problems with Large Chunks:
  1. Internal Fragmentation:
  2. Hot Spots:
  3. Startup Issues:
    • Small configuration files
    • Many clients read at startup
    • Temporary hotspot
    Solution: Batch downloads, higher replication

​
Metadata Design

Why Keep Metadata in RAM?

​
Consistency Without Locks

GFS achieves consistency without distributed locking:

​
Component Interactions

​
Master-Chunkserver Communication

​
Client-Master Communication

​
Key Architectural Decisions

Single Master

Decision: One master, many chunkserversWhy:
  • Simplifies design dramatically
  • Strong metadata consistency
  • Global knowledge for decisions
  • Fast in-memory operations
Mitigation:
  • Minimize master involvement
  • Shadow masters for HA
  • Client caching

Large Chunks

Decision: 64MB chunk sizeWhy:
  • Reduces metadata size
  • Fewer network round trips
  • Better throughput
  • Data locality benefits
Trade-off:
  • Internal fragmentation
  • Potential hot spots
  • Not optimal for small files

Separation of Control/Data

Decision: Metadata through master, data direct to chunkserversWhy:
  • Master not bottleneck for data
  • Scales to GB/s throughput
  • Parallel data transfers
  • Flexible data flow routing
Benefit:
  • Master handles 1000s ops/sec
  • Data path limited only by network

Relaxed Consistency

Decision: Defined consistency, not strictWhy:
  • Higher performance
  • Simpler implementation
  • No distributed transactions
  • Matches application needs
Requirement:
  • Application-level handling
  • Deduplication
  • Validation

​
Interview Questions

Expected Answer:GFS has three main components:
  1. Master (single):
    • Stores all metadata in RAM
    • Manages namespace (files/directories)
    • Tracks chunk locations
    • Makes placement decisions
    • Coordinates system-wide operations
  2. Chunkservers (hundreds/thousands):
    • Store 64MB chunks as Linux files
    • Serve read/write requests
    • Replicate data to other chunkservers
    • Report status to master via heartbeat
  3. Clients (many):
    • Application library
    • Caches metadata
    • Communicates with master for metadata
    • Talks directly to chunkservers for data
Key principle: Control and data flow are separated.
Expected Answer:GFS separates control (metadata) and data flow for scalability:Control Flow (Client ↔ Master):
  • Client asks master for chunk locations
  • Master returns list of replicas
  • Client caches this information
  • Happens only once per chunk/file
Data Flow (Client ↔ Chunkservers):
  • Client communicates directly with chunkservers
  • No master involvement for data transfer
  • Enables massive parallel throughput
  • Master not a bottleneck
Benefits:
  • Master handles thousands of metadata ops/sec
  • Data throughput limited only by network/chunkservers
  • System scales to hundreds of clients at GB/s aggregate
Without separation, master would need to handle all data, becoming an immediate bottleneck.
Expected Answer:GFS write operation has three phases:1. Lease Grant:
  • Client asks master for chunk replicas
  • Master grants lease to one replica (primary)
  • Client caches primary and secondary locations
2. Data Push (decoupled from control):
  • Client pushes data to closest replica
  • Each replica forwards to next in chain
  • Pipelined: forward while receiving
  • All replicas ACK when data buffered in memory
  • Data not yet written to disk!
3. Control Flow:
  • Client sends write command to primary
  • Primary assigns serial number (ordering)
  • Primary applies writes to local disk in order
  • Primary forwards write order to secondaries
  • Secondaries apply in same order
  • Secondaries ACK to primary
  • Primary ACKs success/failure to client
Key Insights:
  • Data flow optimized for network topology
  • Control flow ensures consistent ordering
  • Primary serializes concurrent operations
  • No distributed consensus needed
  • If any replica fails, entire write fails (client retries)
Expected Answer:Chunk locations are NOT stored persistently in master, only in-memory. Reasons:Why Not Persist:
  1. Chunkservers are source of truth:
    • Chunkserver knows what chunks it has (on disk)
    • No risk of inconsistency between master and reality
  2. Dynamic nature:
    • Chunks added/deleted frequently
    • Chunkservers may fail
    • Disks may fail
    • Replication changes chunk locations
  3. Simpler consistency:
    • No need to keep persistent state in sync
    • No risk of master having stale information
    • Master polls chunkservers to rebuild state
How It Works:
  1. Startup: Master polls all chunkservers for chunk list
  2. Ongoing: Heartbeat messages update chunk locations
  3. Changes: Chunkservers report additions/deletions
Benefits:
  • Eliminates entire class of consistency bugs
  • Faster recovery (no persistent state to reload)
  • Simpler implementation
Trade-off:
  • Startup requires polling all chunkservers
  • Acceptable because happens rarely and completes quickly

​
Summary

Key Architecture Takeaways:
  1. Single Master Design: Simplicity and strong consistency for metadata
  2. Separation of Concerns: Control (master) vs. Data (chunkservers) flow
  3. Large Chunks: 64MB chunks reduce metadata and network overhead
  4. In-Memory Metadata: Fast operations, simple design
  5. Leases for Consistency: Primary replica orders operations without consensus
  6. Direct Client-Chunkserver: Data flow doesn’t involve master
  7. Relaxed Consistency: Trade-off for performance, application handles edge cases
  8. Record Append: Atomic concurrent appends without locks

​
Up Next

In Chapter 3: Master Operations, we’ll explore:
  • Namespace management and locking
  • Chunk creation and allocation
  • Replica placement strategies
  • Lease management in detail
  • Garbage collection mechanism
The architecture provides the foundation—now we’ll see how the master orchestrates the entire system.

​
Interview Deep-Dive

Strong Answer:The lease mechanism is how GFS delegates write authority without expensive distributed consensus. When a client wants to write to a chunk, the master grants a 60-second lease to one chunkserver, designating it the “primary.” The primary defines the serial order of all mutations to that chunk. All replicas apply mutations in the same order, ensuring consistency. The lease auto-expires after 60 seconds unless the primary requests extensions via heartbeats.Why leases instead of Paxos or distributed locks? Three reasons. First, simplicity: leases are a single timer-based mechanism, while Paxos requires multiple rounds of messaging for every decision. Second, performance: the lease is granted once and reused for all writes during its lifetime, amortizing the coordination cost. Third, safety against split-brain: if the master loses contact with a primary, it simply waits 60 seconds for the lease to expire before granting a new one. This guarantees that at most one primary exists for any chunk at any time, without needing the complex leader election of consensus protocols.The trade-off is a 60-second recovery window. If a primary crashes, no writes can proceed to that chunk until the lease expires. For Google batch workloads, a 60-second pause was acceptable. For a low-latency transaction processing system, it would not be.Follow-up: What happens if the master itself crashes while a lease is outstanding?The lease continues to be valid because it is time-bound, not dependent on the master being alive. When the master recovers (by loading its checkpoint and replaying the operation log), it polls all chunkservers to discover which leases are still active. Any primary whose lease has not expired continues to serve writes. Any primary whose lease has expired simply stops accepting writes, and the recovered master can grant new leases. This design is elegant because it separates the lease grant (which requires the master) from the lease enforcement (which is purely time-based and works even during master downtime).
Strong Answer:The primary risk is data loss if the master process crashes without persisting recent mutations. GFS mitigates this with an operation log that records every metadata change before acknowledging it to the client. The operation log is replicated to multiple remote machines, so even if the master machine is destroyed, metadata can be recovered from a remote copy.The second risk is memory exhaustion. At 64 bytes per chunk, the master needs about 1GB of RAM per 16 million chunks (about 1PB of data). As the cluster grows, the master eventually runs out of RAM. Google handled this by using prefix compression for the namespace (reducing per-file overhead to about 64 bytes) and by monitoring heap usage as a capacity planning metric.The third risk is recovery time. After a crash, the master must load the latest checkpoint (a compact B-tree representation of the full namespace) and replay all operation log entries since that checkpoint. If the log is long, recovery takes minutes. GFS mitigates this by checkpointing frequently (the checkpoint runs in a background thread without blocking mutations) and by keeping the checkpoint in a format that can be memory-mapped directly.The fourth risk is GC pauses. A master with tens of gigabytes of heap can experience multi-second garbage collection pauses, during which all metadata operations stall. This was a real operational concern at Google and is one reason they eventually moved to Colossus with distributed metadata.Follow-up: If you were designing this today, would you still use in-memory metadata? What alternatives exist?For clusters under 100PB, in-memory metadata is still the right call. Modern servers with 1-2TB of RAM can hold metadata for exabyte-scale clusters. The alternatives are: (1) a distributed metadata store like Bigtable or FoundationDB, which is what Colossus uses — this removes the single-machine RAM limit but adds the complexity of distributed transactions for metadata; (2) a tiered approach where hot metadata is in RAM and cold metadata is on SSD, similar to how modern databases manage buffer pools. I would start with in-memory and plan the migration to distributed metadata as a known future milestone, because the operational simplicity of a single in-memory master saves enormous engineering time in the early years.
Strong Answer:In GFS, control flow (metadata operations like file open, chunk lookup, lease grant) goes through the master, while data flow (actual reads and writes of file content) goes directly between clients and chunkservers. The master never sees a single byte of file data. This separation is critical for three reasons.First, it prevents the master from becoming a bandwidth bottleneck. If every read and write routed through the master, a single machine would need to handle the aggregate I/O bandwidth of the entire cluster — potentially terabytes per second. By keeping the master off the data path, it only needs to handle metadata RPCs, which are small (a few hundred bytes each) and relatively infrequent (clients cache chunk locations).Second, it enables linear scaling of data throughput. Adding more chunkservers adds more aggregate bandwidth because clients talk to chunkservers directly. The master does not need to scale with data throughput, only with metadata operation rate.Third, it simplifies the master implementation. The master does not need high-bandwidth network interfaces, large disk arrays, or complex buffer management. It is essentially a metadata database that fits in RAM.This pattern has become the standard architecture for distributed systems. HDFS uses the same separation (NameNode for metadata, DataNodes for data). Kafka separates its controller from broker data paths. Kubernetes separates the API server (control plane) from kubelet data operations.Follow-up: Are there cases where you would NOT separate control and data flow?Yes. For very small clusters (under 10 nodes) or systems with extremely low throughput, the separation adds unnecessary complexity. A single coordinator that handles both metadata and data is simpler to operate and debug. Also, for systems where every operation requires a metadata decision (like distributed databases with per-row access control), you may want the metadata and data paths to be collocated for lower latency. The separation is a scaling optimization, and like all optimizations, it has a complexity cost that needs to be justified by the scale.