Java-201 RabbitMQ Message Persistence and Queue Index Details

TL;DR

Scenario: Want to understand where RabbitMQ messages are actually stored, why disk grows, why memory explodes, how to adjust parameters.
Conclusion: Core lies in queue index (.idx) + msg_store (.rdq) + ETS mapping + garbage collection/merge strategy coordination.
Output: A reusable storage mental model + key directory/file/parameter locator + common fault quick reference.

RabbitMQ Architecture

Data Storage

RabbitMQ messages can be divided into two types based on durability:

1. Persistent Messages

Messages are immediately written to disk when arriving at queue
A copy is also kept in memory to improve read performance
When memory resources are tight, the in-memory copy is cleared, keeping only disk storage
Suitable for critical business data such as order processing, payment records

2. Transient Messages

By default stored only in memory
When memory pressure reaches threshold (e.g., memory usage exceeds 40%), temporarily dumped to disk
Messages lost after service restart
Suitable for data with high real-time requirements but allowing loss

RabbitMQ’s storage architecture contains two core components:

1. Queue Index

Uses B+ tree structure to store message metadata
Records message location information in storage files
Default storage location: /var/lib/rabbitmq/mnesia/queue-name.idx

2. Message Store

Files actually storing message content
Uses sequential write approach to improve IO performance
Supports message merge and garbage collection mechanism
Default storage location: /var/lib/rabbitmq/mnesia/msg_store_persistent

Queue Index

rabbit_queue_index maintains information about queue’s persisted messages, such as storage location, whether received by consumer, whether acknowledged, etc. Each queue has corresponding index.

Index uses sequential segment files for storage, suffix is .idx, filename starts from 0 and increments
Each segment contains fixed segment_entry_count records, default value is 16384
When reading messages from disk, at least one segment file must be maintained in memory
Be格外 careful when setting queue_index_embed_msgs_below value, a small increase may cause explosive memory growth

Message Store

RabbitMQ’s message store mechanism is implemented using rabbit_msg_store module:

1. Storage Architecture

Uses key-value (Key-Value) storage form, message content written to file in binary
Each virtual host (vhost) has independent storage space, all queues share the same storage area
In cluster environment, each node maintains its own message store instance

2. Storage Type Division

Persistent Storage (msg_store_persistent): Stores messages declared as durable, write flow: message first writes to memory buffer, then persisted to disk via fsync operation
Transient Storage (msg_store_transient): Stores non-persistent messages, uses memory-mapped file (mmap) approach

3. Storage Management Features

Uses file pre-allocation strategy, default single storage file size is 16MB (configurable)
Periodically merges fragmented files through garbage collection mechanism
Persistent storage uses synchronous write strategy to ensure data safety, while transient storage uses async write to improve throughput

Store uses files for storage, suffix is .rdq. All messages processed by store are written to this file in append mode. When file size exceeds specified limit (file_size_limit), the file will be closed and a new file created for new messages. Filename starts from 0 and increments.

Messages (including message header, body, attributes) can be stored directly in index, or in store. Best approach is to store smaller messages in index and larger messages in store. This message size threshold can be configured via queue_index_embed_msgs_below, default is 4096B.

Configuration example (rabbitmq.conf):

## Size in bytes below which to embed messages in the queue index.
# queue_index_embed_msgs_below = 4096
## You can also set this size in memory units
# queue_index_embed_msgs_below = 4kb

If message is smaller than this value, it’s stored in index; if larger, it’s stored in store. Messages larger than this value are stored in .rdq files in msg_store_persistent directory.

When reading messages, first find the storage file based on message ID (msg_id). If file exists and is not locked, directly open file and read message content from specified position. If file doesn’t exist or is locked, request is sent to store for processing.

When deleting messages, only the specified message’s related information is deleted from ETS table, and the storage file and related information are updated. When executing message deletion, messages in file are not immediately deleted, only marked as garbage data.

When a file contains all garbage data, the file can be deleted. When detecting that valid data from two adjacent files can be merged into one file, and the ratio of garbage data size to all files’ data size (at least 3 files must exist) exceeds the set threshold garbage_fraction (default 0.5), garbage collection is triggered to merge these two files.

Merge logic:

Lock these two files
First organize valid data from the front file, then organize valid data from the back file
Write valid data from back file into front file
Update records in message’s ETS table
Delete the back file

Queue Structure

A queue typically consists of rabbit_amqqueue_process and backing_queue. rabbit_amqqueue_process handles protocol-related message processing: accepting messages from producers, delivering messages to consumers, handling message acknowledgments, etc. backing_queue is the specific form and engine of message storage.

rabbit_variable_queue.erl source code in detail defines RabbitMQ Variable Queue’s 4 message storage states:

1. alpha state

Storage: Both message index and content are completely stored in memory
Feature: Provides fastest access speed
Resource consumption: Highest memory usage but lowest CPU consumption

2. beta state

Storage: Message index kept in memory, message content stored on disk
Feature: Moderate memory usage, fast access speed

3. gamma state

Storage: Message index saved in both memory and disk, message content stored on disk
Feature: Further reduced memory usage

4. delta state

Storage: Both message index and content completely stored on disk
Feature: Lowest memory usage

These states dynamically transition based on message access frequency and system resource conditions.

For ordinary queues without priority and mirroring set, backing_queue’s default implementation rabbit_variable_queue internally uses 5 sub-queues Q1, Q2, delta, Q3, Q4 to reflect various message states.

Error Quick Reference

Symptom	Root Cause	Fix
Messages lost after restart	Messages/queue not persisted	Explicitly set durable + persistent message for critical paths
Disk usage continuously grows, doesn’t decrease after deletion/consumption	Deletion is “mark + ETS/index update”, garbage collection conditions must be met for reclamation	Adjust garbage_fraction; perform compaction during off-peak hours
Memory suddenly spikes or OOM	queue_index_embed_msgs_below too large causes more messages “embedded in index”	Lower embed threshold or stratify by queue type
Throughput significantly drops, I/O wait high	fsync/flush overhead of persistent writes	Distribute persistent queues to different disks/nodes
Single large message causes performance jitter	Large message enters store causing heavier disk path	Business side message downsizing; lower embed threshold if necessary
Consumer表现为 “queue not empty but pull slow/jittering”	Messages migrating between Q1/Q2/Delta/Q3/Q4, cold messages in disk state	Increase consumer concurrency or batch ACK; isolate hot and cold queues