Skip to main content

​
Chapter 4: YARN - Yet Another Resource Negotiator

YARN (Yet Another Resource Negotiator) represents a fundamental redesign of Hadoop introduced in Hadoop 2.0. It decouples resource management from data processing, transforming Hadoop from a MapReduce-only system into a general-purpose cluster operating system capable of running diverse workloads.
Chapter Goals:
  • Understand why YARN was created and what problems it solves
  • Master YARN architecture: ResourceManager, NodeManager, ApplicationMaster
  • Learn resource allocation and scheduling algorithms
  • Explore how different frameworks (MapReduce, Spark, Flink) run on YARN
  • Compare YARN with Hadoop 1.x and modern alternatives like Kubernetes

​
Why YARN? The Evolution from Hadoop 1.x

​
Problems with Hadoop 1.x (MRv1)

​
The YARN Solution

Decouple Resource Management from Processing

​
YARN Architecture

​
Core Components

​
ResourceManager Deep Dive

Internal Architecture
FIFO, Capacity, and Fair Schedulers

​
Deep Dive: Hierarchical Queues and Preemption

In large production clusters, YARN is shared by multiple departments, teams, and project types. To manage this complexity, YARN uses Hierarchical Queues.

​
1. Capacity Scheduler: The Organizations Choice

The CapacityScheduler is designed for organizations with strict resource quotas.
  • Guaranteed Capacity: Each queue is guaranteed a percentage of cluster resources.
  • Elasticity: If Queue A is idle, Queue B can use its resources.
  • User Limits: Prevent a single user from hogging all resources in a queue.

​
2. Fair Scheduler: The Interactive Choice

The FairScheduler ensures that all applications get an equal share of resources over time.
  • Fair Share: If two apps are running, they each get 50%.
  • Dominant Resource Fairness (DRF): Handles multi-dimensional resources. If App A is CPU-heavy and App B is Memory-heavy, DRF calculates fairness based on the “dominant” resource each app consumes.

​
3. Preemption: Reclaiming Resources

What happens if Queue A is using its guaranteed 50%, and Queue B (also guaranteed 50%) is currently empty? Queue A will scale up to 100%. If a job is then submitted to Queue B, YARN must reclaim resources from Queue A.
  • Graceful Termination: YARN sends a signal to the AM of the over-allocated containers, asking them to finish up.
  • Hard Kill: If the AM doesn’t release containers within a timeout (e.g., 15 seconds), the RM will forcefully kill those containers to satisfy the guarantee of Queue B.
How Applications Request Containers
Fault Tolerance and Failover

​
NodeManager Deep Dive

Per-Node Resource Management

​
ApplicationMaster Deep Dive

From Launch to Completion
MapReduce on YARN
Spark on YARN

​
Deep Dive: YARN State Machines and Lifecycle Internals

The robustness of YARN comes from its strict state-machine-driven design. Both the ResourceManager and ApplicationMaster operate as complex event-driven state machines.

​
1. ResourceManager Application State Machine

The RM tracks every application from submission to completion. If the RM restarts, it reconstructs this state from the RMStateStore.

​
2. ApplicationMaster Scheduling Lifecycle

The AM’s primary job is to manage the lifecycle of individual tasks within its application.

​
3. The Resource Negotiation Loop (Heartbeat)

The “Negotiation” in YARN happens entirely through the heartbeat mechanism.
  • NM -> RM Heartbeat (NodeStatusUpdater):
    • Reports: “I have 4GB RAM free and 2 vCores.”
    • Receives: “Kill Container X” or “Launch Container Y.”
  • AM -> RM Heartbeat (ApplicationMasterService):
    • Reports: “I need 10 containers with [1GB, 1Core] on Rack /Rack1.”
    • Receives: “Here are 3 containers on Node A, B, and C.”

​
YARN Container Model

​
Container Abstraction

​
Comparing YARN with Alternatives

​
YARN vs Hadoop 1.x

Structural Differences

​
YARN vs Kubernetes

​
Key Takeaways

Remember These Core Insights:
  1. YARN Decoupled Resource Management: ResourceManager knows only about resources, not applications. ApplicationMaster handles app-specific logic.
  2. Containers Not Slots: Flexible resource allocation (any memory/CPU combination) vs rigid slots. This alone improves utilization by 30-40%.
  3. Multi-Framework Platform: YARN enables Spark, Flink, Tez, and more to coexist, not just MapReduce.
  4. Scalability Beyond 10K Nodes: Separation of concerns allows YARN to scale far beyond Hadoop 1.x limits.
  5. High Availability is Built-In: ResourceManager HA and work-preserving recovery make YARN production-ready.
  6. ApplicationMaster Per App: Isolates application failures. One bad app doesn’t affect others.
  7. Data Locality Still Matters: YARN preserves HDFS data locality awareness, critical for performance.
  8. Evolution Path to Cloud: Understanding YARN helps understand Kubernetes, cloud resource management, and container orchestration.

​
Interview Questions

Expected Answer:YARN solves several critical limitations of Hadoop 1.x:1. Tight Coupling to MapReduce:
  • Hadoop 1.x could ONLY run MapReduce jobs
  • YARN supports any distributed application (Spark, Flink, Tez, etc.)
  • Turns Hadoop into a general-purpose cluster OS
2. Scalability Bottleneck:
  • JobTracker handled everything (resource mgmt + scheduling + monitoring)
  • Single JVM couldn’t scale beyond ~4000-5000 nodes
  • YARN: ResourceManager focuses only on resources, scales to 10K+ nodes
3. Poor Resource Utilization:
  • Fixed map/reduce slots led to idle resources (map slots unused while reduce slots busy)
  • YARN: Flexible containers can be allocated for any purpose
  • Utilization improved from ~60% to ~90%
4. No High Availability:
  • JobTracker failure meant all jobs lost
  • YARN: Built-in RM HA with ZooKeeper, work-preserving recovery
Core Innovation: Separation of concerns—resource management (RM) separate from application logic (ApplicationMaster).
Expected Answer:ApplicationMaster (AM) is a per-application coordinator that manages the lifecycle of a single application.Responsibilities:
  1. Resource Negotiation:
    • Calculates resource needs (memory, CPU per task)
    • Sends ResourceRequests to ResourceManager
    • Receives Container allocations
  2. Task Scheduling:
    • Decides which tasks run in which containers
    • Handles data locality (for HDFS-aware apps like MapReduce)
    • Manages task dependencies (map before reduce)
  3. Task Monitoring:
    • Tracks progress of all tasks
    • Detects task failures
    • Requests replacement containers
  4. Failure Handling:
    • Re-launches failed tasks
    • Implements speculative execution
    • Handles node failures
  5. Lifecycle Management:
    • Starts when application begins
    • Unregisters from RM when application completes
    • Exits and releases all resources
Key Insight: AM is application-specific. MapReduce has MRAppMaster, Spark has SparkContext. This allows YARN to support any framework without changing core YARN code.AM Failure: If AM crashes, ResourceManager can restart it (configurable max attempts). On restart, AM can recover state and continue or restart application.
Expected Answer:Data locality is critical for performance—moving computation to data is far cheaper than moving data to computation.YARN’s Data Locality Mechanism:1. ResourceRequest with Locality Preferences:
2. Locality Levels:
  • Node-local (best): Container on same node as HDFS block
  • Rack-local (good): Container on same rack (faster network)
  • Off-rack (acceptable): Any available node
3. Scheduler Considers Locality:
  • Capacity/Fair Schedulers try to satisfy locality first
  • Wait a configured delay for node-local (default: ~3 seconds)
  • If not available, relax to rack-local
  • Last resort: any node
4. Example (MapReduce):
5. Impact on Performance:
  • Node-local: ~10 Gbps (local disk)
  • Rack-local: ~1 Gbps (rack switch)
  • Off-rack: ~500 Mbps (core switch)
Reading 128MB:
  • Node-local: ~1 second
  • Rack-local: ~10 seconds
  • Off-rack: ~20 seconds
Configuration:
Key Takeaway: YARN doesn’t guarantee locality, but makes best effort. Framework (AM) provides preferences, scheduler tries to satisfy them.
Expected Answer:ML workloads have unique requirements that differ from batch MapReduce:Requirements Analysis:
  1. Long-Running Jobs: ML training jobs run for hours/days (vs minutes for MR)
  2. GPU Resources: Need GPU allocation, not just CPU/memory
  3. Gang Scheduling: Distributed training needs all workers to start together
  4. Preemption Concerns: Can’t kill ML job midway (lose expensive training progress)
  5. Priority: Critical experiments should preempt less important ones
  6. Reservation: Reserve resources for scheduled training runs
Design Approach:1. Custom Scheduler (extends CapacityScheduler):
2. GPU Resource Allocation:
3. Gang Scheduling:Problem: Distributed TensorFlow needs 10 workers. If only 7 available, job can’t start.Solution:
4. Checkpoint-Aware Preemption:Traditional preemption: Kill container immediately (lose hours of training!)ML-aware preemption:
5. Queue Structure:
6. Reservation System:For large, planned training jobs:
Trade-offs:Alternative: Use Kubernetes with Kubeflow. K8s has better GPU support, custom resource definitions, and ML-specific operators. Many companies moving ML workloads from YARN to K8s.
Expected Answer:Problem: Application submitted, ResourceManager accepted it, but ApplicationMaster never starts.Debugging Approach:1. Check ResourceManager Logs:
2. Check Queue Status:
3. Check Cluster Capacity:
4. Check AM Resource Requirements:
Common Root Causes:A. All Nodes Full:
B. Queue at Capacity:
C. Max AM Percentage Exceeded:
D. Unhealthy Nodes:
E. Insufficient Resources on Any Single Node:
Diagnostic Commands:
Prevention:
  • Set appropriate AM memory (don’t request 10GB for AM!)
  • Monitor cluster utilization
  • Configure queue limits appropriately
  • Set up alerts for unhealthy nodes
  • Use preemption to free resources for high-priority apps

​
Further Reading

YARN Documentation

Official Apache Hadoop YARN documentation Architecture, configuration, and administration

Hadoop: The Definitive Guide

Tom White - Chapters on YARN Comprehensive YARN coverage

YARN Paper

“Apache Hadoop YARN: Yet Another Resource Negotiator” Vavilapalli et al., 2013

Spark on YARN

Spark documentation on YARN cluster mode Real-world YARN application example

​
Up Next

In Chapter 5: Ecosystem, we’ll explore:
  • Hive: SQL on Hadoop
  • Pig: Data flow language
  • HBase: NoSQL database on HDFS
  • Oozie: Workflow scheduling
  • Kafka: Stream ingestion
  • How these tools integrate with YARN and HDFS
We’ve mastered YARN, Hadoop’s resource management layer. Next, we’ll see how the rich ecosystem of tools builds on HDFS and YARN to provide higher-level abstractions for data processing.

​
Interview Deep-Dive

Strong Answer:The JobTracker had four fundamental problems. First, scalability: the JobTracker handled resource management, job scheduling, task monitoring, and failure recovery all in a single JVM, limiting clusters to about 4,000-5,000 nodes. Second, flexibility: the cluster could only run MapReduce — no Spark, no Flink, no custom frameworks. Third, resource utilization: fixed map and reduce slots meant that when map slots were idle during the reduce phase, those resources were wasted. Typical utilization was 60-70%. Fourth, single point of failure: if the JobTracker crashed, all running jobs were lost with no HA mechanism.YARN solved these by separating concerns. The ResourceManager handles only resource allocation. The per-application ApplicationMaster handles job-specific logic (scheduling, monitoring, recovery). Containers replace fixed slots with flexible resource bundles. This enabled clusters to scale to 10,000+ nodes, run multiple frameworks simultaneously, and achieve 80-90% resource utilization.New problems YARN introduced: First, complexity. Instead of one daemon to understand and debug (JobTracker), operators now have ResourceManager, NodeManager, and per-application ApplicationMasters. Second, queue management. YARN multi-tenancy requires careful queue configuration (Capacity Scheduler or Fair Scheduler) that can be difficult to tune. Misconfigured queues lead to either resource starvation for some teams or wasteful over-provisioning. Third, container overhead. Each container is a separate JVM, and launching thousands of containers per job has non-trivial overhead. Fourth, the ApplicationMaster pattern puts more responsibility on framework developers, making it harder to write new YARN applications compared to writing a MapReduce job.Follow-up: Kubernetes is increasingly used instead of YARN for running Spark and Flink. What does Kubernetes do better?Kubernetes offers better container isolation (cgroups v2, namespaces), a richer ecosystem of tooling (Prometheus, Grafana, Helm), native support for heterogeneous workloads (not just data processing), and cloud-native integration (autoscaling node pools, spot instance support). YARN was designed for data processing on HDFS, while Kubernetes is a general-purpose container orchestrator. The trade-off is that Kubernetes does not have data locality awareness (it does not know where HDFS blocks are), so Spark on Kubernetes relies on remote storage (S3, GCS) rather than local HDFS reads.
Strong Answer:The Capacity Scheduler allocates a fixed percentage of cluster resources to each queue. Queue A gets 40%, Queue B gets 30%, Queue C gets 30%. If Queue A is idle, its capacity can be borrowed by other queues but is reclaimed when Queue A jobs arrive. This model is predictable and easy to reason about for capacity planning.The Fair Scheduler aims to give each active user or queue an equal share of resources. If two jobs are running, each gets 50%. If a third arrives, resources are redistributed to 33% each. This model maximizes utilization and responsiveness but can lead to unpredictable performance during contention.I would choose Capacity Scheduler for multi-tenant production clusters where teams have SLAs and budget allocation. Finance team gets 30%, engineering gets 50%, ad-hoc analytics gets 20%. Each team knows their guaranteed capacity and can plan accordingly. This is the default at most large enterprises.I would choose Fair Scheduler for research or development clusters where workloads are unpredictable and fairness is more important than guaranteed capacity. It provides better interactive performance because short jobs get resources quickly instead of waiting in a queue.In practice, both schedulers have converged in functionality. Capacity Scheduler now supports preemption and dynamic resource allocation, while Fair Scheduler supports hierarchical queues and capacity limits. The choice often comes down to organizational preference and existing configuration.Follow-up: What happens when a high-priority job needs resources that are currently occupied by lower-priority jobs?YARN supports preemption, where the scheduler kills containers from lower-priority jobs to free resources for higher-priority ones. Preemption is aggressive and can waste work if a long-running task is killed near completion. YARN mitigates this by first trying to reclaim resources through natural container completion (waiting for tasks to finish) and only resorting to killing containers if the high-priority job has waited too long. Application-level checkpointing (saving intermediate state) can reduce the cost of preemption by allowing killed tasks to resume from a checkpoint rather than restarting from scratch.
Strong Answer:My debugging process follows a top-down approach, starting from the coarsest metrics and drilling down.Step 1: Check YARN ResourceManager UI. Look at queue utilization — is the application waiting in the queue (scheduling delay) or actually running? If it is pending, the problem is resource availability, not the application itself. Check if the queue has available capacity and whether other jobs are hogging resources.Step 2: Check the Spark application UI. Look at the stage timeline — which stages are slow? Identify whether the bottleneck is in a specific stage (data skew, expensive UDF) or across all stages (resource problem). Check task distribution within each stage — if one task takes 10x longer than others, you have data skew.Step 3: Check executor metrics. Are executors fully utilized (CPU and memory)? If memory is high and there are frequent GC pauses, the executors need more memory. If CPU is low, the job might be I/O bound — check disk and network throughput.Step 4: Check the shuffle. If the slow stage is a shuffle-dependent stage (anything after a groupBy, join, or repartition), look at shuffle read/write sizes. Large shuffle sizes (100GB+) indicate that data is being moved across the network. Check for skewed keys by looking at the task metrics in the Spark UI — one task reading significantly more shuffle data than others indicates key skew.Step 5: Check HDFS or S3 access patterns. If input splits are from HDFS, check data locality (Spark UI shows locality level). If most tasks are “ANY” locality, data is being read remotely. If reading from S3, check if the request rate is being throttled.Follow-up: You found data skew causing one reducer to process 100x more data than others. How do you fix it?Three approaches depending on the operation. For joins with a skewed key: use a broadcast join if the smaller table fits in memory (eliminates the shuffle entirely), or salt the skewed key by appending a random number (0-N) and replicating the other table N times to match. For aggregations with a skewed key: use a two-phase aggregation — first aggregate with salted keys (distributing the work), then aggregate the partial results with the original key. For sorts: if ordering is not critical, repartition with a hash function that distributes the skewed key across multiple partitions.