CHUNKSERVER FAILURE HANDLING
───────────────────────────

Scenario: cs-12 crashes

Detection:
─────────

T+0:    cs-12 crashes
T+30s:  Master expects heartbeat, none arrives
T+60s:  Master expects heartbeat, none arrives
T+90s:  Master declares cs-12 dead
        (3 missed heartbeats)

Master Action:
─────────────

1. Mark cs-12 as dead:
   dead_servers.add(cs-12)

2. Scan all chunks:
   for chunk in all_chunks:
       if cs-12 in chunk.replicas:
           chunk.replicas.remove(cs-12)
           if len(chunk.replicas) < 3:
               schedule_re_replication(chunk)

3. Identify affected chunks:
   • 10,000 chunks on cs-12
   • All now under-replicated

4. Prioritize re-replication:
   • 1 replica: URGENT
   • 2 replicas: NORMAL
   • Already covered above

5. Execute re-replication:
   (covered in re-replication section)


Client Impact:
─────────────

Reads from cs-12:
• Fail immediately (connection error)
• Client retries with different replica
• Transparent to application
• Slight latency spike

Writes to cs-12:
• Primary (cs-12) is dead
• Lease expires (60 seconds max)
• Master grants lease to different replica
• Client retries, succeeds


Recovery if cs-12 Comes Back:
─────────────────────────────

Scenario: cs-12 was network partition, not dead

T+300s: cs-12 reconnects

cs-12 → Master: Heartbeat with chunk list

Master checks:
─────────────

For each chunk cs-12 reports:
1. Is version stale?
   → Mark for garbage collection

2. Is chunk already re-replicated?
   → Now have 4 replicas
   → Delete one (rebalancing)

3. Is chunk still under-replicated?
   → Count cs-12's copy
   → May cancel pending re-replication


False Positive Handling:
───────────────────────

Network partition caused false positive:

• cs-12 was alive, just unreachable
• Master re-replicated chunks
• Now have extra copies
• Rebalancing cleans up

Cost of false positive:
• Extra network bandwidth
• Extra disk space temporarily
• Acceptable (safety first)

DISK FAILURE HANDLING
────────────────────

Scenario: Disk fails on cs-5

Detection:
─────────

Method 1: Read Error
───────────────────

Client reads chunk 0x1a2b from cs-5
cs-5 attempts read from disk
→ I/O error from disk

cs-5 → Master:
  "Chunk 0x1a2b is unreadable (I/O error)"

cs-5 → Client:
  "ERROR: Cannot read chunk"

Method 2: Checksum Mismatch
──────────────────────────

cs-5 reads block from disk
Computes checksum
Compares with stored checksum
→ MISMATCH (silent corruption)

cs-5 → Master:
  "Chunk 0x1a2b is corrupted"

cs-5 → Client:
  "ERROR: Corruption detected"

Method 3: Background Scrubbing
──────────────────────────────

cs-5 background task verifies chunks
Detects corruption before client reads

cs-5 → Master:
  "Chunk 0x1a2b is corrupted"


Master Response:
───────────────

1. Mark cs-5's replica as corrupted:
   chunk_replicas[0x1a2b].mark_corrupted(cs-5)

2. Schedule re-replication:
   re_replication_queue.add(
       chunk=0x1a2b,
       priority=HIGH
   )

3. Select source (good replica):
   source = pick_random([cs-12, cs-23])
   (Other replicas of 0x1a2b)

4. Select destination:
   destination = pick_server(
       low_load=True,
       different_rack=True
   )

5. Execute re-replication:
   (standard process)

6. After new replica confirmed:
   Master → cs-5: "Delete chunk 0x1a2b"


Client Handling:
───────────────

Client receives error from cs-5
Client retries with different replica (cs-12)
cs-12 serves data successfully
Transparent to application


Multiple Disk Failures:
──────────────────────

Scenario: Disk fails on cs-5 (has 1000 chunks)

cs-5 → Master:
  "Chunks [0x1111, 0x2222, ..., 1000 total] failed"

Master:
• Marks all 1000 as corrupted on cs-5
• Schedules 1000 re-replications
• High priority (data at risk)
• Executes over hours


Entire Chunkserver Disk Fails:
─────────────────────────────

If all chunks on cs-5 fail:
• Equivalent to chunkserver failure
• All chunks re-replicated
• cs-5 may be taken out of service
• Human intervention to replace disk

NETWORK PARTITION HANDLING
─────────────────────────

Scenario: Network switch fails, isolating Rack A

Partition:
─────────

┌─────────────────────┐
│  Master + Most      │
│  of cluster         │
│  (Rack B, C, D)     │
└─────────────────────┘
          ✗
          ✗  Network partition
          ✗
┌─────────────────────┐
│  Rack A             │
│  (Isolated)         │
│  cs-1, cs-2, cs-3   │
└─────────────────────┘


Master Perspective:
──────────────────

Rack A chunkservers miss heartbeats
Master declares them dead
Schedules re-replication for all chunks


Rack A Perspective:
──────────────────

Cannot reach master
Leases expire (60 seconds)
Cannot serve writes (no primary)
Can still serve reads (from cache)


When Partition Heals:
────────────────────

T+0:    Partition begins
T+90s:  Master declares Rack A dead
T+120s: Master begins re-replication
T+300s: Partition heals
        Rack A reconnects

Rack A → Master: Heartbeats with chunk lists

Master discovers:
────────────────

1. Rack A servers were alive (false positive)
2. Some chunks already re-replicated (4 replicas now)
3. Some chunks in progress
4. Some chunks not yet started

Master reconciles:
─────────────────

• Cancel pending re-replications
• Keep extra replicas temporarily
• Rebalancing removes extras later
• Update chunk location maps
• Resume normal operations


Split-Brain Prevention:
──────────────────────

Can Rack A serve writes during partition?

NO! Here's why:
──────────────

1. Leases expire after 60 seconds
2. Cannot renew lease (no master contact)
3. Cannot grant new leases
4. Cannot become primary
5. Writes fail

Primary authority requires master contact
→ No split-brain possible

Clients:
• Writes fail (lease expired)
• Retry → master redirects to working replicas
• Reads succeed (from Rack A or other racks)


Cost of Partition:
─────────────────

• Write unavailable for affected chunks
  (during partition)
• Unnecessary re-replication
  (bandwidth, storage waste)
• Extra cleanup after healing
• Acceptable (safety first)

MASTER FAILURE HANDLING
──────────────────────

Scenario: Primary master crashes

Impact:
──────

• Cannot create files
• Cannot delete files
• Cannot grant leases
• Cannot get chunk locations (if not cached)
• WRITES BLOCKED
• Reads continue (if metadata cached)


Detection:
─────────

T+0:    Master crashes
T+10s:  Monitoring service detects
        (heartbeat failure)


Failover Process:
────────────────

T+15s:  External coordination service
        (e.g., Chubby lock service)
        triggers failover

T+20s:  Promote shadow master:

1. Stop accepting read requests
2. Apply any pending log entries
3. Flush in-memory state
4. Switch to write mode:
   • Enable lease grants
   • Enable chunk creation
   • Enable writes

T+30s:  Update DNS entry:
   gfs-master.google.com → new_master_ip

T+45s:  Clients detect change:
   • Cached master IP fails
   • DNS lookup gets new IP
   • Retry operations

T+60s:  System fully operational


During Failover (T+0 to T+60s):
───────────────────────────────

Reads:
• Shadow masters serve reads
• Slightly stale (seconds old)
• Most clients have cached metadata anyway

Writes:
• Fail (no primary master)
• Clients retry
• Eventually succeed when new master ready

New File Operations:
• Fail during failover
• Succeed after new master ready


Data Loss:
─────────

Operation log replicated → NO DATA LOSS

• All mutations in operation log
• Replicated before client sees success
• New master has full log
• Replays and continues


Recovery Details:
────────────────

New master startup:
(See "Fast Recovery" section)

1. Load checkpoint
2. Replay log
3. Poll chunkservers
4. Resume operations

Total: ~60 seconds


If Shadow Also Fails:
────────────────────

Both primary and shadow crash:

• Start new master from scratch
• Load latest checkpoint
• Replay full operation log
• Poll all chunkservers
• Takes longer (~few minutes)
• But still recoverable
• No data loss


Disaster: All Masters Gone
──────────────────────────

Extremely rare (datacenter failure):

• Operation logs in multiple datacenters
• Load from remote datacenter
• Start master in different location
• Recovery takes longer (10-30 minutes)
• Still no data loss (logs persistent)


Client Behavior During Failover:
────────────────────────────────

Built-in retry logic:

def gfs_operation_with_retry(op):
    max_retries = 10
    for i in range(max_retries):
        try:
            return op()
        except MasterUnreachable:
            sleep(exponential_backoff(i))
            continue  # Retry
    raise Exception("Master unavailable")

Transparent to most applications
Brief unavailability acceptable