RabbitMQ Architecture and Exchange Routing Details

TL;DR

• Scenario: Microservices decoupling, async peak cutting, broadcast notifications, need to “deliver messages to the right queue”
• Conclusion: Exchange determines routing strategy; direct exact, topic wildcard, fanout broadcast, headers multi-condition but more CPU-intensive
• Output: A version matrix + an error quick reference card, directly for troubleshooting and implementation reference

---

Version Matrix

Feature	RabbitMQ 3.x	RabbitMQ 4.x
Classic Mirrored Queues	✅ Compatible	❌ Removed
Quorum Queues	✅	✅ Recommended
AMQP 1.0	Plugin	Native Support

Verification Note: Classic Mirrored Queues are deprecated and removed since RabbitMQ 4.0; HA recommends Quorum Queues/Streams

---

RabbitMQ Architecture

Basic Introduction

RabbitMQ is an open-source message broker developed by Rabbit Technologies Ltd, using Mozilla Public License open-source license.

Industry Applications

• Internet industry: Used for service decoupling and async communication in microservices architectures
• Telecom industry: Originally designed for reliable communication between telecom systems
• Financial industry: Processing trading messages, payment notifications, etc.
• IoT: Communication with sensor devices
• Traditional enterprise IT systems: System integration, data synchronization, etc.

Core Features

1. High Reliability:

   • Supports message persistence
   • Provides publish confirmation mechanism
   • Supports Transaction and Publisher Confirm

2. Easy Scalability:

   • Supports cluster deployment
   • Can easily add nodes
   • Supports mirrored queues for high availability

3. Rich Features:

   • Multiple exchange types: Direct, Topic, Fanout, Headers
   • Message TTL (time-to-live)
   • Dead letter queues
   • Priority queues
   • Message tracing

Technical Implementation

1. AMQP Protocol:

   • RabbitMQ implements AMQP 0-9-1 protocol
   • AMQP (Advanced Message Queuing Protocol) is an application layer standard protocol providing unified message services

2. Erlang Language:

   • Developed using Erlang OTP platform
   • Erlang naturally supports high concurrency and distribution
   • Provides good fault tolerance mechanism

3. Multi-protocol Support:

   • Native AMQP protocol support
   • Also supports MQTT (via plugin)
   • STOMP (via plugin)
   • HTTP (via management plugin)

Typical Application Scenarios

1. Async processing: e.g., sending email/SMS notifications after user registration
2. Application decoupling: e.g., separating order system from inventory system
3. Traffic peak cutting: Handling sudden high concurrency scenarios
4. Log processing: Centrally collecting distributed system logs
5. Message broadcasting: e.g., system configuration change notifications

---

Exchange

RabbitMQ commonly used exchange types and their application scenarios:

fanout (Broadcast Mode)

• Feature: Routes messages to all bound queues, ignoring routing key
• Application Scenario: Suitable for scenarios requiring broadcast messages, such as system notifications, log distribution
• Example: An order creation event needs to simultaneously notify inventory system, logistics system, and points system

Working Principle:

1. Message producer sends message to fanout exchange
2. Exchange does not check message’s routing key
3. Exchange sends copies of message to each bound queue
4. Each bound queue receives exactly the same message

Implementation Example:

# Declare fanout exchange
channel.exchange_declare(exchange='logs', exchange_type='fanout')

# Bind queues
channel.queue_bind(exchange='logs', queue='queue1')
channel.queue_bind(exchange='logs', queue='queue2')

# Publish message
channel.basic_publish(exchange='logs', routing_key='', body='Message')

Notes:

• Messages are copied multiple times, which may affect system performance
• Not suitable for scenarios with strict message processing order requirements
• Consumers need to handle possibly duplicate messages

---

direct (Direct Mode)

• Feature: Routes messages to queues based on exact matching of routing key
• Application Scenario: Suitable for scenarios requiring precise routing, such as specific business processing
• Example: Orders with successful payment route to “order.payment.success” queue, failed payment route to “order.payment.failed” queue

Routing Rules:

1. Exact Matching Mechanism: Message is only routed to a queue when the message’s Routing Key exactly matches the queue’s Binding Key.

   • If Routing Key is “order.created”, only queues with Binding Key “order.created” receive this message
   • “order” will not match “order.created”, “order.*” will not match either

2. Multiple Queue Binding: Multiple queues can bind to the same direct exchange using the same Binding Key. In this case, message is simultaneously routed to all matching queues, implementing simple broadcast functionality.

Typical Application Scenarios:

• Order processing system: Can use different Routing Keys like “order.created”, “order.paid” to distinguish different types of order messages
• Log collection system: Use Routing Keys like “log.error”, “log.warning” to classify different levels of logs
• Task distribution system: Distribute to specific processing queues based on task type using different Routing Keys

Performance Advantage: Due to exact matching algorithm, direct exchange routing efficiency is high, suitable for performance-critical scenarios.

Differences from Other Exchange Types:

• Compared to fanout exchange (broadcasts to all queues), direct is selective routing
• Compared to topic exchange (supports pattern matching), direct only supports exact matching
• Compared to headers exchange (based on message header matching), direct uses Routing Key matching

Note: If no queue’s Binding Key matches the message’s Routing Key, the message will be directly discarded unless an alternate exchange is configured.

---

topic (Topic Mode)

• Feature: Supports wildcard matching of routing keys (supports * and # wildcards), more flexible than direct
• Application Scenario: Suitable for scenarios requiring classified message processing

Key Name Structure:

• Both RoutingKey and BindingKey use ”.” separated multi-segment naming
• Example: “stock.usd.nyse” or “weather.asia.china.shanghai”

Special Wildcards:

1. Asterisk ”*”:

   • Exactly matches one word segment
   • Example: BindingKey “stock.*.nyse” can match:
     • “stock.usd.nyse”
     • “stock.eur.nyse”
   • But cannot match:
     • “stock.nyse” (missing one segment)
     • “stock.usd.future.nyse” (one extra segment)

2. Hash ”#”:

   • Matches zero or more word segments
   • Example: BindingKey “weather.#” can match:
     • “weather” (zero subsequent segments)
     • “weather.asia”
     • “weather.asia.china”
     • “weather.asia.china.shanghai”
   • Is the most relaxed matching mode

Typical Application Scenarios:

• Log-level processing:
  • BindingKey “syslog.*.error” can receive all subsystem error logs
  • BindingKey “syslog.auth.#” can receive all authentication-related logs at all levels
• Geographic location message routing:
  • “location.#.temperature” can receive temperature data for any regional level
  • “location.country.*.humidity” can receive humidity data for any city under a country

Usage Notes:

• Wildcards must exist as independent segments, cannot appear within words
• Valid BindingKey examples: “.stock”, “market.#”, “order..completed”
• Invalid BindingKey examples: “stck” (wildcard not at segment start), “market..#” (consecutive wildcards)

---

headers (Headers Mode)

• Feature: Routes based on message header attributes, ignoring routing key
• Application Scenario: Suitable for special scenarios requiring routing based on message attributes
• Matching Rules:
  • x-match=all: Must match all header key-value pairs
  • x-match=any: Only needs to match any one header key-value pair

Workflow:

1. Binding Phase:

   • When creating a queue and binding it to headers exchange, need to specify a set of key-value pairs as matching conditions
   • Can set matching mode to “all” (all conditions must be met) or “any” (any condition satisfied)

2. Message Routing Phase:

   • When producer sends message, need to set corresponding attributes in message header
   • Exchange extracts all attributes from message header
   • Compares message header attributes with key-value pairs specified when queue was bound
   • When matching conditions are met, message is routed to corresponding queue

Implementation Example:

# Bind queue with matching conditions
channel.queue_bind(
    queue='high_priority_queue',
    exchange='headers_exchange',
    arguments={'x-match':'all', 'priority':'high', 'region':'north'}
)

# Set headers when sending message
properties = pika.BasicProperties(
    headers={'priority': 'high', 'region': 'north'}
)
channel.basic_publish(
    exchange='headers_exchange',
    routing_key='',  # headers exchange ignores routing key
    body=message,
    properties=properties
)

Performance Notes:

• Headers exchange needs to parse each message’s headers and perform multi-condition matching
• Matching operation consumes more CPU resources
• Performance significantly degrades when queue count is high
• In actual production environments, unless there are special requirements, using direct or topic exchange is recommended

Applicable Scenarios:

• Scenarios requiring complex routing based on multiple conditions
• Scenarios where routing conditions may change frequently
• Scenarios where message attributes better describe routing needs than routing keys

---

Error Quick Reference

Symptom	Root Cause	Diagnosis	Fix
Producer sent message but queue is empty	direct/topic routing key doesn’t match binding key; or no queue bound at all	Management console check exchange→bindings; verify routing_key vs binding_key	Add bindings; unify naming conventions; add alternate-exchange/mandatory+return if needed
Message “disappears” without error	No mandatory set; message discarded when no matching binding (default behavior)	Enable publisher confirms/returns on producer side; check return callback	mandatory=true and handle return; or configure alternate exchange
Fanout causes severe throughput drop	Broadcast causes message replication N copies, queue/disk/network amplification	Check if enqueue rate and queue backlog scale linearly with subscriber count	Reduce subscriber count; split topics; change to topic precise distribution; rate limiting and batching
Headers routing latency increases, CPU high	Each message needs to parse headers and do multi-condition matching	perf/CPU flame graph; compare CPU at same throughput with direct/topic	If routing key can be used, don’t use headers; simplify conditions; reduce binding count
HA config fails upgrade or queue unavailable	Still using Classic Mirrored Queues, removed since 4.0	Check queue type/policy; verify if classic mirrored	Migrate to Quorum Queues/Streams (follow official migration path)
Consumer sees duplicate consumption	At-least-once delivery semantics + improper ack timing/timeout retry	Check consumer ack and retry logs; check redelivered flag	Business idempotency; correct ack; use retry queue/dead letter queue governance