TL;DR

  • Scenario: Coexisting with traditional IBM MQ, need open source, operatable, scalable, consistency/reliability
  • Conclusion:
    • RabbitMQ: Suits “reliability-first business decoupling”
    • RocketMQ: Suits “transaction/order/delay messages”
    • Kafka: Suits “data pipeline/log/stream processing”
  • Output: Selection comparison table + deployment considerations + error quick reference

Middleware Selection Principles

Selection Criteria

  1. Open Source: Must be open source to ensure transparency and autonomy
  2. Popularity: Recent trend technology, active community
  3. Core Function Requirements:
    • Message transmission reliability
    • Cluster support capability
    • Excellent performance

RabbitMQ

Overview: Based on AMQP protocol, lightweight and easy to deploy.

Advantages

  1. Lightweight and Efficient:

    • Small installation package (tens of MB)
    • Second-level startup
    • Docker support

  2. Flexible Routing:

    • Direct Exchange: Exact routing key match
    • Fanout Exchange: Broadcast to all bound queues
    • Topic Exchange: Supports wildcard matching

  3. Multi-language Support:

    • Official support for Java, .NET, Python
    • Community support for 30+ languages

Disadvantages

  1. Message Backlog Issue: Throughput drops 50%+ when queue exceeds 100,000 messages
  2. Performance Limitations:
    • Persistent messages: ~50,000-80,000 TPS
    • Non-persistent messages: ~100,000-150,000 TPS
  3. Erlang Tech Stack: Plugin development requires Erlang knowledge

RocketMQ

Overview: Distributed message middleware open-sourced by Alibaba, based on Java.

Typical Application Scenarios

  1. Ordered message processing (e.g., order status changes)
  2. Distributed transactional messages (e.g., e-commerce systems)
  3. Real-time stream computing
  4. Message push notifications

Advantages

  1. Comprehensive Features:

    • Publish/Subscribe and P2P modes
    • Message ordering, transactional messages, scheduled messages, message tracking

  2. Developer Friendly:

    • Java implementation, clear source code structure
    • SPI extension mechanism support

  3. Performance:

    • Average response time < 3ms
    • Single machine: 10W+ TPS
    • Cluster mode: Million-level throughput

Disadvantages

  1. Ecosystem Integration: Limited Prometheus integration
  2. Operations Complexity: Multi-replica sync configuration complex

Kafka

Overview: Distributed stream processing platform designed for high throughput.

Advantages

  1. High Reliability: Multi-replica mechanism ensures no data loss
  2. Excellent Stability: Validated by LinkedIn, Uber; MTBF up to 99.99%
  3. Rich Features: Exactly-once semantics, transactional messages, message replay
  4. Ecosystem Compatibility: Deep integration with Hadoop, Spark, Flink
  5. Excellent Performance:
    • Single machine: 100,000+ TPS
    • Cluster mode: Million-level TPS

Technical Features

  1. Scalable Architecture:

    • Partition-based horizontal scaling
    • Consumer Group mechanism achieves linear consumption scaling

  2. Persistent Storage:

    • Default 7-day retention
    • Sequential read/write + page cache technology

  3. Replica and Fault Tolerance:

    • Multi-replica support (usually 3 replicas)
    • ISR mechanism ensures data consistency

Overall Comparison

FeatureRabbitMQRocketMQKafka
LanguageErlangJavaJava/Scala
ProtocolAMQPCustomCustom
TPS50K-150K100K+100K-2M
LatencyLowVery LowMedium
OrderingSupportedSupportedSupported
TransactionSupportedStrongSupported
EcosystemGoodModerateExcellent

Selection Suggestions

  1. Choose RabbitMQ:

    • Reliability over performance
    • Complex routing needs
    • Medium throughput requirements (tens of thousands TPS)

  2. Choose RocketMQ:

    • Need transactional message support
    • Need ordered messages
    • Finance/trading systems

  3. Choose Kafka:

    • High throughput requirements (hundreds of thousands+ TPS)
    • Data pipeline/log processing scenarios
    • Need big data ecosystem integration

Error Quick Reference

RabbitMQ Issues

SymptomRoot CauseFix
Throughput drops, latency jitterQueue backlog, memory pressureCheck queue depth; reduce backlog through queue splitting, TTL+DLQ

RocketMQ Issues

SymptomRoot CauseFix
Send timeoutNameServer routing inconsistent, Broker pressureEnsure NameServer HA; optimize Broker parameters

Kafka Issues

SymptomRoot CauseFix
Consumer Lag risesInsufficient partitions, downstream slowIncrease partitions; optimize batch processing; control rebalance
ISR shrinksBroker I/O bottleneckUpgrade disk/network; adjust replica parameters