Message Queue vs. Publish/Subscribe
Message Queue
- Communication Pattern: Point-to-point (one sender to one receiver)
- Message Handling: Each message is processed by exactly one consumer
- Use Case: Task distribution among workers (work queues)
- Example: A task queue where multiple workers pull jobs but each task is completed only once
- Characteristics:
- Messages remain in the queue until processed
- Once a message is consumed, it's removed from the queue
- Good for workload distribution across multiple consumers
Publish/Subscribe (Pub/Sub)
- Communication Pattern: One-to-many (one publisher to multiple subscribers)
- Message Handling: Each message is delivered to all active subscribers
- Use Case: Event broadcasting, notifications, real-time updates
- Example: News feed updates where all subscribers receive the same content
- Characteristics:
- Messages are broadcast to all subscribers
- Multiple consumers receive copies of the same message
- Supports event-driven architecture and decoupled systems
Apache Kafka Intro
- Primarily a pub/sub system but can be used for queuing
- Uses topic partitions to balance between queue and pub/sub patterns
- Consumer groups allow for both queue-like and pub/sub behaviors
- Messages are persisted for configurable retention periods