Message Queues and Event Streaming
Last reviewed: August 2026
Overview
Section titled “Overview”An easy way to understand it
Section titled “An easy way to understand it”A message queue is like a restaurant’s order slip. When the front of house (producer) puts up a slip, it piles up while the kitchen (consumer) is busy, and gets processed in order once there’s capacity. The front of house doesn’t need to know the kitchen’s state.
Event streaming is like a radio broadcast. The station (producer) transmits once, and multiple listeners (consumers) may each tune in at different times. If it’s recorded (retained), it can also be listened to again later.
Why it’s needed — the problem with synchronous calls
Section titled “Why it’s needed — the problem with synchronous calls”If the order service calls payment → inventory → notification synchronously:
- If even one is slow, everything is delayed
- If even one fails, the order fails
- During a traffic surge, cascading failures occur
Adding a message queue: the order completes immediately, and the rest is processed at its own pace. Failures are isolated.
What was used on-premises
Section titled “What was used on-premises”| Cloud service | On-premises equivalent |
|---|---|
| SQS, Service Bus Queue | IBM MQ (MQ Series), RabbitMQ, ActiveMQ |
| MSK, Event Hubs | Apache Kafka (self-managed) |
| SNS, Event Grid | RabbitMQ Exchange (fanout), TIBCO |
| EventBridge, Eventarc | ESB (Enterprise Service Bus) — though lighter-weight than an ESB |
Message queue vs. event streaming
Section titled “Message queue vs. event streaming”Direct (synchronous) calls between microservices increase coupling and propagate failures. Message queues and event streaming decouple inter-service communication asynchronously, providing loose coupling, load buffering, and failure isolation.
| Aspect | Message queue | Event streaming |
|---|---|---|
| Model | Producer → queue → consumer (1:1 or fan-out) | Producer → topic → multiple consumers (Pub/Sub) |
| Message retention | Deleted after consumption | Can be re-read during the retention period |
| Order guarantee | FIFO option | Order guaranteed within a partition |
| Suitable for | Work queues, async processing, load distribution | Event sourcing, real-time analytics, log collection |
Comparison of vendor services
Section titled “Comparison of vendor services”| Area | AWS | Azure | Google Cloud | OCI |
|---|---|---|---|---|
| Message queue | SQS | Service Bus Queue | Cloud Tasks | OCI Queue |
| Pub/Sub | SNS | Service Bus Topic | Pub/Sub | OCI Streaming |
| Event routing | EventBridge | Event Grid | Eventarc | OCI Events |
| Streaming (Kafka-compatible) | MSK | Event Hubs (Kafka protocol compatible) | Pub/Sub + Dataflow | OCI Streaming (Kafka compatible) |
What to choose when
Section titled “What to choose when”| Requirement | Recommendation |
|---|---|
| Simple work queue (async processing, retries) | SQS, Service Bus Queue, Cloud Tasks, OCI Queue |
| Event fan-out (1:N notification) | SNS, Service Bus Topic, Pub/Sub |
| Event-driven architecture (routing, filtering) | EventBridge, Event Grid, Eventarc |
| High-volume real-time streaming (logs, clickstream) | MSK/Kafka, Event Hubs, Pub/Sub, OCI Streaming |
| Event sourcing (history replay needed) | Kafka (MSK), Event Hubs (Capture) |
| Vendor-neutral (multi-cloud) | Apache Kafka (self-managed or Confluent Cloud) |
Common mistakes
Section titled “Common mistakes”- Confusing message queues with event streaming — Adopting Kafka when a 1:1 work queue is needed, or choosing SQS when event replay is required, results in a mismatched architecture.
- Not configuring a Dead Letter Queue (DLQ) — If failed messages retry endlessly, the queue gets clogged and normal messages also stop being processed.
- Using a standard queue when message ordering is required — Standard queues don’t guarantee order. If order matters, choose a FIFO queue or partition-key-based streaming.
Checklist
Section titled “Checklist”- Have you chosen the message pattern (1:1 queue vs. 1:N fan-out vs. streaming) to fit your requirements?
- Have you configured a Dead Letter Queue and retry policy (max attempts, backoff)?
- Have you confirmed there’s no message loss on consumer failure (at-least-once guarantee)?