Choosing a message broker: Kafka vs SQS vs RabbitMQ

Every broker selection deck we've reviewed for a client starts with a feature matrix: throughput, latency, managed vs self-hosted, ecosystem maturity. That matrix produces the wrong answer more often than it produces the right one, because it treats Kafka, SQS, and RabbitMQ as three implementations of the same idea. They're not. They encode three different beliefs about what a message is for, and the only question that actually predicts whether you'll be happy with your choice in eighteen months is: what do you need to be true when a consumer is broken for six hours?

The six-hour-outage test

With SQS, a broken consumer means messages pile up in the queue, invisible after their visibility timeout expires, redelivered to whoever picks them up next, and eventually routed to a dead-letter queue after the max-receive-count is hit. The queue doesn't care about order across messages and it doesn't remember what already happened. That's fine for a job queue. It's a liability for anything that needs to reconstruct state from history.

With Kafka, a broken consumer just falls behind. The topic keeps every message for its retention window, seven days by default, and the consumer resumes from its last committed offset the moment it comes back. Nothing is lost, nothing is redelivered out of order within a partition, and you can point a brand-new consumer group at offset zero and replay the entire history to rebuild a projection. That capability is the entire reason to choose Kafka. If you don't need it, Kafka is overkill.

RabbitMQ sits in between and gets punished for it. Classic mirrored queues have no durable log concept: once a message is acked and removed, it's gone. Quorum queues (the Raft-based queue type since RabbitMQ 3.8, now the recommended default over classic mirroring) add durability and predictable failover, but you're still working with queues that drain, not a log you can rewind. RabbitMQ's real strength is routing: exchanges, bindings, and header-based fanout give you message-shape flexibility that neither Kafka nor SQS offers cleanly.

Ordering is a tax, not a checkbox

Kafka gives you ordering only within a partition, and only if you key messages consistently. Get the partition key wrong (using an internal request ID instead of the entity ID, say) and you'll ship an application with silent ordering bugs that nobody notices until two events for the same customer race each other in production. SQS FIFO queues give you strict ordering per message group, capped at 300 transactions per second per API call without batching (3,000 with batching), which rules them out for high-throughput pipelines outright. Standard SQS queues give you no ordering guarantee at all and will occasionally deliver a message twice. RabbitMQ gives you strict FIFO per queue as long as you have exactly one consumer; add a second consumer for throughput and ordering is gone unless you shard by key yourself, which is just reinventing Kafka partitions with worse tooling.

Operational burden is a staffing decision

Self-managed Kafka is a distributed system with its own failure modes: partition rebalances that stall consumers, consumer lag that silently grows until someone notices a downstream report is stale, disk pressure from retention misconfiguration. Managed Kafka (MSK, Confluent Cloud) removes the broker operations but not the schema, partitioning, and consumer-group design decisions, which is where most Kafka outages actually originate. SQS has effectively zero operational surface: no capacity planning, no patching, no partition strategy. RabbitMQ falls in between: lighter to run than Kafka, but mirrored and quorum queue behavior under network partition is a real thing you need someone who has read the clustering docs to reason about, not something to leave to defaults.

When each one is the wrong choice

Kafka is the wrong choice for a task queue with no replay requirement: you'll pay for partition management and consumer-group complexity to get FIFO-per-key behavior SQS gives you for free. SQS is the wrong choice the moment two services need to see the same event, or when a new service needs to backfill from history: fan-out means duplicating the message into SNS or writing it twice, and there is no rewind. RabbitMQ is the wrong choice for event sourcing or analytics pipelines where the log itself is the source of truth: it was built to move messages, not to be a system of record for them.

If we're advising a client, we ask what breaks worse: losing an event, or losing the ability to prove what happened. That answer, not the throughput column, is what should pick the broker.