> ## Documentation Index > Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt > Use this file to discover all available pages before exploring further. # RabbitMQ Internals > How RabbitMQ actually works - Erlang/OTP, message flow, clustering, and quorum queues # RabbitMQ Internals Deep Dive > **If you love understanding how things actually work, this chapter is for you. If you just want to send and receive messages, feel free to skip ahead. No judgment.** This chapter takes you inside RabbitMQ. We will explore how Erlang enables RabbitMQ's reliability, understand the complete message flow, and demystify clustering and high availability. This knowledge is what allows you to build truly resilient messaging systems. *** ## Why Internals Matter Understanding RabbitMQ internals helps you: * **Design resilient systems** that survive failures * **Troubleshoot production issues** when messages go missing * **Choose the right queue type** for your use case * **Ace interviews** where messaging internals are valued * **Tune for performance** when throughput matters *** ## Erlang: The Secret Weapon RabbitMQ is built on **Erlang/OTP**, and this choice shapes everything about its architecture. ### Why Erlang? Erlang was designed by Ericsson in 1986 for telecom switches - systems that needed: * **99.999% uptime** (5 nines) * **Hot code upgrades** without stopping * **Massive concurrency** (millions of connections) * **Fault isolation** (failures do not cascade) These are exactly what a message broker needs. ### Erlang Processes (Not OS Processes) Erlang has its own lightweight process model: ``` OS Process: ┌─────────────────────────────────────────────────────────────────┐ │ Erlang VM (BEAM) │ │ │ │ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │ │ │ P1 │ │ P2 │ │ P3 │ │ P4 │ │ P5 │ │ P6 │ ... millions │ │ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ │ │ │ │ Each Erlang process: │ │ - Has its own heap (garbage collected independently) │ │ - Has a mailbox (message queue) │ │ - Weighs ~2KB initially │ │ - Can be supervised (auto-restart on crash) │ └─────────────────────────────────────────────────────────────────┘ ``` **In RabbitMQ**: * Each connection = Erlang process * Each channel = Erlang process * Each queue = Erlang process * Supervision trees automatically restart failed components ### The OTP Framework OTP (Open Telecom Platform) provides patterns for building reliable systems: ``` Supervision Tree: ┌───────────────────┐ │ rabbit_sup │ (Root supervisor) └─────────┬─────────┘ │ ┌─────────────────────┼─────────────────────┐ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Connection │ │ Queue │ │ Exchange │ │ Supervisor │ │ Supervisor │ │ Supervisor │ └──────┬───────┘ └──────┬───────┘ └──────────────┘ │ │ ┌───┴───┐ ┌───┴───┐ ▼ ▼ ▼ ▼ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │Conn1│ │Conn2│ │ Q1 │ │ Q2 │ └─────┘ └─────┘ └─────┘ └─────┘ ``` If a queue process crashes, its supervisor restarts it. If multiple children crash, supervisor may restart the whole subtree. This "let it crash" philosophy is why RabbitMQ is remarkably stable. *** ## Message Flow: From Producer to Consumer Let us trace a message through RabbitMQ: ### 1. Publishing ``` Producer RabbitMQ │ │ │──── AMQP Connection ──────▶│ │ │ │──── Channel.Open ─────────▶│ (Create channel process) │ │ │──── Basic.Publish ────────▶│ │ routing_key="orders" │ │ exchange="shop" │ │ body= │ │ │ │ ┌──────▼──────┐ │ │ Exchange │ │ │ (lookup) │ │ └──────┬──────┘ │ │ │ ┌──────────────┴──────────────┐ │ │ Routing (bindings lookup) │ │ └──────────────┬──────────────┘ │ │ │ ┌──────▼──────┐ │ │ Queue(s) │ │ │ (enqueue) │ │ └─────────────┘ ``` ### 2. Exchange Routing Each exchange type has different routing logic: | Exchange | Routing Logic | Use Case | | ----------- | ------------------------------- | ----------------------- | | **Direct** | Exact routing key match | Point-to-point, RPC | | **Fanout** | Broadcast to all bound queues | Notifications, events | | **Topic** | Pattern matching on routing key | Selective subscriptions | | **Headers** | Match on message headers | Complex routing rules | **Topic Pattern Matching**: ``` Routing Key: "orders.us.new" Binding: "orders.#" -> MATCH (# = zero or more words) Binding: "orders.*.new" -> MATCH (* = exactly one word) Binding: "orders.eu.*" -> NO MATCH (eu != us) Binding: "*.us.*" -> MATCH ``` ### 3. Queue Storage Messages in a queue can be: * **In memory**: Fast, lost on restart * **On disk**: Durable, survives restart * **Both**: For persistent messages with in-memory cache ``` Queue Process: ┌─────────────────────────────────────────────────────────────────┐ │ │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ In-Memory Queue │ │ │ │ [Msg1] [Msg2] [Msg3] [Msg4] [Msg5] ... │ │ │ └──────────────────────────────────────────────────────────┘ │ │ │ │ │ │ If persistent and memory pressure │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ Disk Queue (Mnesia + segment files) │ │ │ │ /var/lib/rabbitmq/mnesia/rabbit@host/queues/... │ │ │ └──────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` ### 4. Consumer Delivery ``` RabbitMQ Consumer │ │ │ │ ┌──────────────────────▼─────────────────────┐ │ │ Queue Process │ │ │ │ │ │ Prefetch check: consumer has capacity? │ │ │ │ │ │ │ │ Yes │ │ │ ▼ │ │ │ Dequeue message │ │ │ │ │ │ │ ▼ │ │ │ Mark as unacked (in flight) │ │ │ │ │ │ └───────┼─────────────────────────────────────┘ │ │ │ │───── Basic.Deliver ──────────────────────▶│ │ │ │ │ Process │ │ │◀───── Basic.Ack ─────────────────────────│ │ │ ┌───────▼─────────────────────────────────────┐ │ │ Remove from unacked, message complete │ │ └─────────────────────────────────────────────┘ │ ``` *** ## AMQP Protocol Deep Dive AMQP (Advanced Message Queuing Protocol) is the wire protocol RabbitMQ implements. ### Connection and Channels ``` TCP Connection (expensive): ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Channel 1 ─────────▶ Queue A operations │ │ │ │ Channel 2 ─────────▶ Queue B operations │ │ │ │ Channel 3 ─────────▶ Queue C operations │ │ │ │ (Channels are lightweight, multiplexed over one connection) │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` **Best Practice**: One connection per application, one channel per thread. ### Message Acknowledgments ``` auto_ack=true (fire and forget): Producer ──▶ Broker ──▶ Consumer │ └── Message deleted immediately auto_ack=false (manual acknowledgment): Producer ──▶ Broker ──▶ Consumer │ │ │ ▼ │ Process message │ │ │◀── ack ───┘ │ └── Message deleted after ack If consumer dies before ack: │ └── Message requeued (redelivered=true) ``` ### Publisher Confirms How to know if RabbitMQ received your message: ``` Publisher: Broker: enable confirms mode │ ├───── publish msg 1 ─────────▶ received, stored │◀───── confirm (1) ────────── ACK │ ├───── publish msg 2 ─────────▶ received, stored │◀───── confirm (2) ────────── ACK │ ├───── publish msg 3 ─────────▶ FAILED (disk full?) │◀───── nack (3) ────────────── NACK │ └── handle failure (retry, alert, etc.) ``` *** ## Queue Types: Classic vs Quorum vs Stream RabbitMQ offers multiple queue types for different needs: ### Classic Queues (Original) ``` Classic Queue: ┌─────────────────────────────────────────────────────────────────┐ │ Single Erlang Process │ │ - Fast, simple │ │ - Single point of failure (unless mirrored) │ │ - Messages stored in Mnesia + disk segments │ └─────────────────────────────────────────────────────────────────┘ Classic Mirrored Queue (HA): ┌───────────────────┐ ┌───────────────────┐ │ Node 1 │ │ Node 2 │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ │ Queue │ │ │ │ Mirror │ │ │ │ (master) │──┼─────┼──│ (replica) │ │ │ └─────────────┘ │ │ └─────────────┘ │ └───────────────────┘ └───────────────────┘ Problem: Synchronous replication is slow and complex ``` ### Quorum Queues (Recommended for HA) ``` Quorum Queue (Raft-based): ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ Node 1 │ │ Node 2 │ │ Node 3 │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ │ QQ Member │ │ │ │ QQ Member │ │ │ │ QQ Member │ │ │ │ (leader) │◀─┼─────┼──│ (follower) │──┼─────┼──│ (follower) │ │ │ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │ └───────────────────┘ └───────────────────┘ └───────────────────┘ - Raft consensus (majority must agree) - Automatic leader election - Data safety first, then performance - Recommended for production HA ``` **Quorum = Majority**: For 3 nodes, quorum is 2. For 5 nodes, quorum is 3. ### Streams (Kafka-like) ``` Stream (RabbitMQ 3.9+): ┌─────────────────────────────────────────────────────────────────┐ │ Append-only log │ │ │ │ [Offset 0] [Offset 1] [Offset 2] [Offset 3] [Offset 4] ... │ │ │ │ - Messages retained by time/size (not deleted on consume) │ │ - Multiple consumers can read same messages │ │ - Consumers can seek to any offset │ │ - High throughput for fan-out patterns │ └─────────────────────────────────────────────────────────────────┘ ``` | Feature | Classic | Quorum | Stream | | -------------------- | ------------- | ---------------- | ---------------- | | **HA Model** | Mirror (sync) | Raft (consensus) | Replication | | **Message Deletion** | On ack | On ack | Retention policy | | **Ordering** | Per queue | Per queue | Offset-based | | **Use Case** | Simple queues | Critical HA | Log/replay | *** ## Clustering RabbitMQ nodes form a cluster to share metadata and enable HA. ### What is Shared in a Cluster | Component | Shared? | Notes | | ------------------------------ | ------- | -------------------------------- | | **Users, vhosts, permissions** | Yes | Stored in Mnesia, replicated | | **Exchanges** | Yes | Metadata replicated to all nodes | | **Bindings** | Yes | Metadata replicated to all nodes | | **Queue metadata** | Yes | Name, durability, arguments | | **Queue messages** | No | Only on node hosting the queue | ``` Cluster (3 nodes): ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Node 1 Node 2 Node 3 │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Queue A │ │ Queue B │ │ Queue C │ │ │ │ (messages) │ │ (messages) │ │ (messages) │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ All nodes know: │ │ - Queue A exists on Node 1 │ │ - Queue B exists on Node 2 │ │ - Queue C exists on Node 3 │ │ │ └─────────────────────────────────────────────────────────────────┘ Client can connect to any node - requests proxied to queue owner. ``` ### Cluster Formation ```bash theme={null} # On node 2, join node 1's cluster rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl join_cluster rabbit@node1 rabbitmqctl start_app # Verify cluster status rabbitmqctl cluster_status ``` ### Partition Handling Network partitions are the bane of distributed systems: ``` Normal: [Node1] ◀────────▶ [Node2] ◀────────▶ [Node3] Partition: [Node1] ◀────────▶ [Node2] X [Node3] Partition A Partition B ``` RabbitMQ partition handling modes: | Mode | Behavior | Risk | | ---------------- | ------------------------- | ----------------------------- | | `ignore` | Both partitions continue | Split brain, data divergence | | `pause_minority` | Minority partition pauses | Safe, may reduce availability | | `autoheal` | Restart nodes in minority | Data loss possible | **Recommendation**: Use `pause_minority` for most cases. *** ## Flow Control and Backpressure RabbitMQ protects itself from being overwhelmed. ### Credit Flow Erlang processes use credit flow between each other: ``` Connection Process ──[credits]──▶ Channel Process ──[credits]──▶ Queue Process When credits run out: - Upstream process blocks - Wait for credits to replenish - This propagates backpressure to publishers ``` ### Memory and Disk Alarms ``` Memory watermarks: - vm_memory_high_watermark = 0.4 (40% of RAM) - When exceeded: producers blocked, consumers continue Disk watermarks: - disk_free_limit = 50MB (or {mem_relative, 1.0}) - When exceeded: all publishing blocked Check status: $ rabbitmqctl status ... Alarms: memory (blocking) ... ``` *** ## Interview Deep Dive Questions **Answer**: Three things must be durable: 1) Queue declared with durable=true (survives restart), 2) Messages published with persistent=true (written to disk), 3) Publisher confirms enabled (know when written). For HA, use quorum queues (Raft consensus) or classic mirrored queues. Even with all this, messages can be lost if acked by consumer but not processed. **Answer**: Mirrored queues use synchronous replication (all mirrors must sync before ack), which is slow and complex. Quorum queues use Raft consensus (majority must agree), which is safer and handles partitions better. Quorum queues are the recommended approach for HA in RabbitMQ 3.8+. Mirrored queues are deprecated. **Answer**: Prefetch (QoS) limits unacknowledged messages per consumer. Default is unlimited (dangerous). With prefetch=10, consumer gets up to 10 messages before acking. Benefits: 1) Load balancing - slow consumers get fewer messages, 2) Memory control - limits messages in flight, 3) Fairness - no consumer hogs the queue. Set via basic.qos(prefetch\_count=N). **Answer**: Classic queues: messages on that node are unavailable until node recovers (unless mirrored). Quorum queues: if leader fails, Raft elects new leader from followers, queue continues serving (with majority). Cluster: other nodes detect failure, client connections to dead node drop, clients should reconnect to surviving nodes. **Answer**: Messages are ordered within a queue (FIFO). But: 1) With multiple consumers, messages are distributed - no ordering across consumers, 2) Requeued messages (nack with requeue) go to front or back (configurable), 3) Dead letter exchange changes order. For strict ordering: single consumer, or partition by key (consistent hashing exchange), or use streams. **Answer**: RabbitMQ: complex routing (topic, headers), request-reply (RPC), task queues where messages are deleted after processing, lower latency for small messages. Kafka: high-throughput event streaming, replay capability, longer retention, log aggregation, when consumers need to read same messages multiple times. RabbitMQ streams blur this line. *** ## Monitoring and Debugging ### Management Plugin ```bash theme={null} # Enable management UI rabbitmq-plugins enable rabbitmq_management # Access at http://localhost:15672 # Default: guest/guest (only localhost) ``` ### Key Metrics to Watch | Metric | Warning Sign | | -------------------- | --------------------------------------- | | **Queue depth** | Growing constantly = consumers too slow | | **Unacked messages** | High count = consumers not acking | | **Memory usage** | Approaching watermark = blocking soon | | **File descriptors** | Approaching limit = connection failures | | **Disk space** | Approaching limit = publishing blocked | ### Debugging Commands ```bash theme={null} # List queues with message counts -- the first command to run when "messages are stuck" # messages_ready = waiting to be delivered, messages_unacknowledged = delivered but not acked # If messages_ready is growing: consumers are too slow or disconnected # If messages_unacknowledged is growing: consumers are receiving but not acking (bug or stall) rabbitmqctl list_queues name messages messages_ready messages_unacknowledged # List connections -- check for stale or blocked connections # "blocking" state means the connection is paused due to a memory or disk alarm rabbitmqctl list_connections name state channels # List consumers -- verify your consumers are actually connected # An empty list when you expect consumers means they crashed or lost connection rabbitmqctl list_consumers # Check for alarms -- memory and disk alarms block publishers # If publishers are mysteriously stuck, this is often the cause rabbitmqctl status | grep -A5 "Alarms" # Trace messages (DEVELOPMENT ONLY -- this generates enormous output in production) # Shows every message publish and deliver in real time rabbitmqctl trace_on ``` **Production gotcha**: Running `rabbitmqctl trace_on` on a production broker can generate gigabytes of log data per minute and significantly impact performance. Use it only in development or on a test broker. For production debugging, use the management UI's message rates and queue depths, or enable per-queue tracing on a single low-traffic queue. *** ## Key Takeaways 1. **Erlang/OTP is the foundation** - lightweight processes, supervision trees, "let it crash" 2. **AMQP is the protocol** - connections hold channels, channels hold operations 3. **Exchanges route, queues store** - understand the four exchange types 4. **Durability requires three things** - durable queue, persistent message, publisher confirm 5. **Quorum queues for HA** - Raft consensus beats mirrored queues 6. **Streams for replay** - append-only log, Kafka-like semantics 7. **Prefetch controls load** - always set it, never use unlimited 8. **Clustering shares metadata** - messages stay on their queue's node *** Ready to build reliable messaging patterns? Next up: [RabbitMQ Patterns](/courses/devops-tools/rabbitmq-patterns) where we will implement work queues, pub/sub, and RPC.