NAMENODE MEMORY (~150 bytes per block)
──────────────────────────────────────

For 100 million blocks (12.5 PB with 128MB blocks):
• Block metadata: ~15 GB
• File metadata: ~5 GB
• Total: ~20 GB (fits in single server RAM)


1. Namespace (File System Tree)
───────────────────────────────

InodeFile:
┌────────────────────────────────┐
│ Path: /user/hadoop/data.txt    │
│ Owner: hadoop                  │
│ Group: supergroup              │
│ Permissions: 644               │
│ Replication: 3                 │
│ Block Size: 128MB              │
│ Modification Time: 1642531200  │
│ Access Time: 1642535000        │
│ Blocks: [blk_1, blk_2, blk_3]  │
└────────────────────────────────┘

InodeDirectory:
┌────────────────────────────────┐
│ Path: /user/hadoop/            │
│ Owner: hadoop                  │
│ Permissions: 755               │
│ Children: [input, output]      │
└────────────────────────────────┘


2. Block Map (Block → Locations)
────────────────────────────────

BlockInfo:
{
  blockId: blk_1234567890,
  size: 134217728,  // 128MB
  generationStamp: 1001,
  replicas: [
    {datanode: DN1, state: FINALIZED},
    {datanode: DN3, state: FINALIZED},
    {datanode: DN5, state: FINALIZED}
  ]
}

Quick lookups:
• Given block ID → where is it stored?
• Given file → what blocks compose it?


3. DataNode Descriptors
───────────────────────

DataNodeDescriptor:
{
  nodeId: "DN1",
  hostname: "datanode1.example.com",
  ipAddress: "10.0.1.5",
  xferPort: 50010,
  infoPort: 50075,
  ipcPort: 50020,

  capacity: 4398046511104,      // 4TB
  dfsUsed: 2535301169152,       // 2.3TB
  remaining: 1862745341952,     // 1.7TB

  blocks: Set<BlockInfo>,
  lastHeartbeat: 1642531200,
  lastBlockReport: 1642520400
}

Enables:
• Load balancing decisions
• Replication target selection
• Cluster health monitoring

FSIMAGE (Namespace Snapshot)
────────────────────────────

Checkpoint of entire namespace:
• All directories and files
• File → block mappings
• Permissions, ownership
• Replication factors
• Block sizes

Format: Protobuf (compact binary)
Size: Gigabytes for large clusters
Created: Periodically by Secondary NameNode

Example:
/fsimage/fsimage_0000000000012345678


EDIT LOG (Transaction Log)
──────────────────────────

All namespace mutations since last fsimage:

┌──────────────────────────────────────┐
│ TxID: 12345678                        │
│ Op: OP_ADD                            │
│ Path: /user/hadoop/data.txt           │
│ Replication: 3                        │
│ Timestamp: 1642531200                 │
├──────────────────────────────────────┤
│ TxID: 12345679                        │
│ Op: OP_ALLOCATE_BLOCK                 │
│ Path: /user/hadoop/data.txt           │
│ BlockID: blk_1234567890               │
│ GenStamp: 1001                        │
├──────────────────────────────────────┤
│ TxID: 12345680                        │
│ Op: OP_CLOSE                          │
│ Path: /user/hadoop/data.txt           │
└──────────────────────────────────────┘

Enables:
• Fast recovery (replay edits)
• Durability (survive crashes)
• High availability (standby replays)


RECOVERY PROCESS
────────────────

On NameNode startup:

1. Load latest fsimage
   ↓
2. Replay edit log entries
   ↓
3. Merge to create current namespace
   ↓
4. Receive block reports from DataNodes
   ↓
5. Enter safe mode (read-only)
   ↓
6. Verify block replication meets threshold
   ↓
7. Exit safe mode (read-write)

Time: Minutes to hours (depends on size)


CHECKPOINT PROCESS
──────────────────

Secondary NameNode periodically:

1. Downloads fsimage and edits from NameNode
2. Loads fsimage into memory
3. Applies edit log transactions
4. Saves new merged fsimage
5. Uploads to NameNode
6. NameNode replaces old fsimage
7. Starts new edit log

Why? Keeps edit log from growing unbounded
Frequency: Hourly or when edits reach threshold

    HDFS HA ARCHITECTURE WITH QJM
    ──────────────────────────────

             ┌────────────────┐          ┌────────────────┐
             │   Active NN    │          │   Standby NN   │
             │ (Read/Write)   │          │  (Read-only)   │
             └───────┬────────┘          └───────┬────────┘
                     │                           │
                     │   ┌───────────────────┐   │
                     └──▶│   JournalNodes    │◀──┘
                         │   (Quorum of 3, 5)│
                         └───────────────────┘
                                   │
               ┌───────────────────┴───────────────────┐
               │                                       │
        ┌──────▼──────┐                         ┌──────▼──────┐
        │    ZKFC     │◀───────(ZooKeeper)──────▶│    ZKFC     │
        └─────────────┘                         └─────────────┘

HDFS FEDERATION VIEW
────────────────────

  NameService 1          NameService 2          NameService 3
 ┌─────────────┐        ┌─────────────┐        ┌─────────────┐
 │  NN1 (HA)   │        │  NN2 (HA)   │        │  NN3 (HA)   │
 │  /user/*    │        │  /data/*    │        │  /tmp/*     │
 └──────┬──────┘        └──────┬──────┘        └──────┬──────┘
        │                      │                      │
        └───────────┬──────────┴───────────┬──────────┘
                    │                      │
             ┌──────▼──────┐        ┌──────▼──────┐
             │  DataNode   │        │  DataNode   │
             │ (Pool 1,2,3)│        │ (Pool 1,2,3)│
             └─────────────┘        └─────────────┘

TRIGGERS FOR RE-REPLICATION
──────────────────────────

1. DataNode Failure
   DN3 dies → all its blocks under-replicated
   NameNode detects (no heartbeat)
   Initiates re-replication immediately

2. Disk Failure
   DN reports disk error
   Blocks on that disk → under-replicated
   Re-replicate from other replicas

3. Replication Factor Increase
   User increases replication: 3 → 5
   NameNode creates 2 more replicas per block

4. Corrupt Block Detected
   Checksum mismatch
   Mark block corrupt
   Re-replicate from good replicas


PRIORITY QUEUE
──────────────

Blocks prioritized by replication level:

Priority 1: 0 replicas (data loss imminent!)
Priority 2: 1 replica (one failure = loss)
Priority 3: < replication factor

Process highest priority first


RE-REPLICATION PROCESS
──────────────────────

1. NameNode identifies under-replicated blocks
   ↓
2. Selects source DataNode (has good replica)
   ↓
3. Selects target DataNode (where to copy)
   Considers:
   • Rack diversity
   • Available space
   • Current load
   ↓
4. Sends command: DN_source → copy block to DN_target
   ↓
5. Source initiates transfer to target
   ↓
6. Target stores block
   ↓
7. Target reports to NameNode
   ↓
8. NameNode updates block map


THROTTLING
──────────

Limit re-replication bandwidth:
• Default: 2 concurrent transfers per DataNode
• Prevents network saturation
• Avoids impacting client requests
• Configurable: dfs.datanode.max.transfer.threads

During large-scale failures:
• May take hours to re-replicate all blocks
• Cluster degraded but operational
• Prioritization ensures critical data first

DECOMMISSIONING WORKFLOW
────────────────────────

Scenario: Remove DN3 from cluster (hardware upgrade)

1. Administrator marks DN3 for decommission
   Edit: hdfs-site.xml (exclude file)
   Command: hdfs dfsadmin -refreshNodes

2. NameNode puts DN3 in "Decommissioning" state
   DN3 still accepts heartbeats
   No new blocks assigned to DN3

3. NameNode identifies blocks only on DN3
   Creates list of all blocks to replicate elsewhere

4. Re-replication begins
   For each block on DN3:
   • Select target DataNode
   • Copy block to target
   • Update block map

5. Monitor progress
   Command: hdfs dfsadmin -report
   Shows: "Decommissioning in progress: 45% complete"

6. All blocks copied
   DN3 state: "Decommissioned"
   DN3 has zero unique blocks

7. Safe to power down DN3
   No data loss
   Cluster continues normally


WHY DECOMMISSION (Don't just power off!)
────────────────────────────────────────

Without decommissioning:
✗ Immediate under-replication
✗ NameNode thinks DN3 is dead
✗ Emergency re-replication triggered
✗ Network saturation
✗ Potential data loss if multiple failures

With decommissioning:
✓ Gradual, controlled process
✓ No data loss risk
✓ Network-friendly
✓ Clean removal


TIME ESTIMATE
─────────────

Depends on:
• Amount of data on node
• Network bandwidth
• Cluster load
• Number of concurrent decommissions

Typical: Hours to days for heavily-loaded nodes

THE BALANCING PROBLEM
─────────────────────

Over time, cluster becomes imbalanced:

DN1: 90% full  ▓▓▓▓▓▓▓▓▓░
DN2: 45% full  ▓▓▓▓░░░░░░
DN3: 80% full  ▓▓▓▓▓▓▓▓░░
DN4: 30% full  ▓▓▓░░░░░░░

Causes:
• New nodes added (start empty)
• Uneven write patterns
• Some nodes have larger disks
• Decommissioned nodes

Problems:
✗ Some nodes fill up first
✗ New writes fail (no space)
✗ Uneven read load
✗ Inefficient resource use


HDFS BALANCER
─────────────

Standalone tool that redistributes blocks:

Command:
$ hdfs balancer -threshold 10

Meaning:
• Target: All nodes within 10% of average utilization
• Average: 61.25% full
• Goal: Each node between 51.25% - 71.25%


BALANCER ALGORITHM
──────────────────

1. Calculate cluster average utilization
   Total used / Total capacity = 61.25%

2. Identify over-utilized nodes (> threshold)
   DN1: 90% (over by 28.75%)
   DN3: 80% (over by 18.75%)

3. Identify under-utilized nodes (< threshold)
   DN2: 45% (under by 16.25%)
   DN4: 30% (under by 31.25%)

4. For each over-utilized node:
   Select blocks to move
   Choose under-utilized destination
   Initiate block transfer

5. Repeat until balanced or max iterations


CONSTRAINTS
───────────

Balancer obeys replication policy:
✓ Maintains rack diversity
✓ Doesn't reduce replicas below factor
✓ Respects rack awareness

Throttling:
• Default: 10 MB/s per DataNode
• Configurable: dfs.datanode.balance.bandwidthPerSec
• Runs in background, low priority

Safety:
✓ Only moves replicas (doesn't delete)
✓ Verifies checksums after move
✓ Updates NameNode metadata atomically
✓ Can be stopped safely anytime


BEST PRACTICES
──────────────

When to run:
→ After adding new nodes
→ When utilization becomes skewed
→ During maintenance windows (low load)

How often:
→ Weekly or monthly (depends on write rate)
→ Not constantly (overhead)

Monitoring:
→ Check progress: hdfs balancer -h
→ Watch cluster balance: hdfs dfsadmin -report

SCENARIO: DN3 Fails During Write
────────────────────────────────

Original pipeline:
Client ──▶ DN1 ──▶ DN3 ──▶ DN4
                   ❌ (fails)


Step 1: Detect Failure
──────────────────────

DN1 doesn't receive ACK from DN3
DN1 timeout (configurable: 60s)
DN1 reports to Client


Step 2: Rebuild Pipeline
────────────────────────

Client:
• Removes DN3 from pipeline
• Current replicas: DN1, DN4 (only 2!)
• Continues with reduced pipeline

New pipeline:
Client ──▶ DN1 ──▶ DN4


Step 3: Resume Writing
──────────────────────

Client resends packets not ACKed:
• Keeps track of unacknowledged packets
• Resends from last successful packet
• No data loss


Step 4: Inform NameNode
───────────────────────

Client ──▶ NameNode
    "DN3 failed during write of blk_1234567890"

NameNode:
• Marks DN3 as dead (or suspect)
• Notes block is under-replicated


Step 5: Post-Write Re-Replication
─────────────────────────────────

After file closed:
NameNode schedules re-replication
Block copied from DN1 or DN4 to new DN (e.g., DN5)
Final replicas: [DN1, DN4, DN5]


MULTIPLE FAILURES
─────────────────

If 2 DataNodes fail:
Still have 1 replica
Continue writing
Re-replicate after close

If all 3 fail:
Write fails
Client retries with new pipeline
NameNode selects different DataNodes


BLOCK VERSIONS
──────────────

Each attempt creates new generation stamp:

Attempt 1: blk_1234567890, gen_stamp: 1001
Attempt 2: blk_1234567890, gen_stamp: 1002

Prevents confusion with partial writes
Old generation stamps eventually garbage collected


CLIENT-SIDE ROBUSTNESS
──────────────────────

Client maintains:
• Packet queue (unacknowledged)
• Maximum retries
• Timeout configurations
• Error counters

If too many failures:
→ Report to application
→ Application decides retry or fail

CHECKSUM VERIFICATION
─────────────────────

Write Path:

1. Client computes checksums
   Per 512 bytes of data

2. Sends data + checksums in packet

3. Each DataNode:
   • Receives packet
   • Verifies checksums
   • Writes data to disk
   • Stores checksums separately
   • Forwards to next DataNode

4. If checksum mismatch:
   • DataNode reports error
   • Client excludes DataNode from pipeline
   • Rebuilds pipeline without bad node


ATOMIC BLOCK CREATION
─────────────────────

Blocks created atomically:

During write:
• Block in "under construction" state
• Not visible to readers
• Stored in temporary location

After close:
• Block finalized
• Moved to final location
• Made visible to readers
• Generation stamp finalized

If crash before finalization:
• Incomplete blocks deleted
• No partial data visible


LEASE MECHANISM
───────────────

Prevents concurrent writes:

Client ──▶ NameNode: "open for write"
NameNode grants exclusive lease (60s)

Client must renew lease:
• Heartbeat every 30s
• While file open

If client crashes:
• Lease expires after 60s
• NameNode recovers file
• Discards incomplete last block

Another client tries to write same file:
NameNode: "Lease already held, wait or fail"


PIPELINE ACK
────────────

Guarantees all replicas written:

Client sends packet
↓
DN1 writes ──▶ DN3 writes ──▶ DN4 writes
                                ↓
                             DN4 ACKs
                                ↓
                        DN3 ACKs (after DN4)
                                ↓
                    DN1 ACKs (after DN3)
                                ↓
                            Client receives ACK

Only when all replicas confirm:
→ Client considers packet written
→ Moves to next packet


FAILURE ATOMICITY
─────────────────

Either all replicas succeed or none:

Success: All ACKs received
→ Packet committed

Partial failure: Some ACK, some don't
→ Remove failed DataNodes
→ Resend packet
→ Continue with reduced pipeline

Complete failure: No ACKs
→ Retry packet
→ If persistent, fail entire block
→ Start new block

APPEND WORKFLOW
───────────────

Use case: Log aggregation
Multiple writers append to same file


Step 1: Request Append
──────────────────────

Client ──▶ NameNode: "append /var/log/app.log"

NameNode:
• Checks permissions
• Verifies file exists
• Grants lease to client
• Returns last block location

NameNode ──▶ Client:
    Last block: blk_999, size: 64MB
    Locations: [DN2, DN5, DN8]


Step 2: Append Data
───────────────────

If last block < 128MB:
• Append to existing block
• Establish pipeline with existing replicas
• Continue from last offset

If last block full (128MB):
• Allocate new block
• Normal write pipeline


Step 3: Pipeline Synchronization
────────────────────────────────

Critical: All replicas must agree on block state

Client ──▶ DN2, DN5, DN8:
    "What's your last confirmed offset for blk_999?"

Responses:
DN2: 67108864 bytes (64MB)
DN5: 67108864 bytes (64MB)
DN8: 67108864 bytes (64MB)

All agree → Continue from 64MB

If mismatch:
DN2: 64MB
DN5: 64MB
DN8: 63MB ← Out of sync!

Resolution:
• Use minimum confirmed offset
• Truncate longer replicas
• Restart from 63MB


Step 4: Continue Writing
────────────────────────

Same as normal write:
• Packet streaming
• ACK protocol
• Checksum verification


CONCURRENCY CONTROL
───────────────────

Only ONE writer at a time:

Client A holds lease → Can append
Client B tries to append → Blocked until lease released

Lease duration: 60 seconds
Renewable while writing

If Client A crashes:
• Lease expires
• NameNode recovers file
• Client B can now append


USE CASES
─────────

Good for:
✓ Log aggregation
✓ Audit trails
✓ Sequential data streams

Not for:
✗ Random writes
✗ Frequent small appends (overhead)
✗ Multiple concurrent writers (lease contention)


LIMITATIONS
───────────

Performance:
⚠ Append slower than fresh write
⚠ Pipeline synchronization overhead
⚠ Potential for replica divergence

Recommendation:
→ For high-volume logs, write to new files periodically
→ Append for occasional additions only