Message Queue vs. Publish/Subscribe

Message Queue

• Communication Pattern: Point-to-point (one sender to one receiver)
• Message Handling: Each message is processed by exactly one consumer
• Use Case: Task distribution among workers (work queues)
• Example: A task queue where multiple workers pull jobs but each task is completed only once
• Characteristics:
  • Messages remain in the queue until processed
  • Once a message is consumed, it's removed from the queue
  • Good for workload distribution across multiple consumers

Publish/Subscribe (Pub/Sub)

• Communication Pattern: One-to-many (one publisher to multiple subscribers)
• Message Handling: Each message is delivered to all active subscribers
• Use Case: Event broadcasting, notifications, real-time updates
• Example: News feed updates where all subscribers receive the same content
• Characteristics:
  • Messages are broadcast to all subscribers
  • Multiple consumers receive copies of the same message
  • Supports event-driven architecture and decoupled systems

Apache Kafka Intro

• Primarily a pub/sub system but can be used for queuing
• Uses topic partitions to balance between queue and pub/sub patterns
• Consumer groups allow for both queue-like and pub/sub behaviors
• Messages are persisted for configurable retention periods