In streams, we have the producer-consumer model:

• Producers generate/gather/aggregate data
• Consumers process data from producers
  • Note that a consumer could be a producer, e.g. processing raw data into aggregated use

There are different methods of using the producer-consumer model:

• Push/callback method
  • Producer simply sends data to consumer over some medium (e.g. connection)
  • Callback when action ends
  • With this method, may not be able to process all incoming data
    • e.g. cannot fit all data into buffer
• Push/poll method
  • Consumer requests data from producer
  • Producer saves data as it generates it
  • Data from producer is eventually given to consumer

Consumers may be consuming data from multiple producers, and a producer might need to send data to multiple consumers (many-many relationship).

In order to scale, we use middleware that coordinates streaming - data broker.

• Broker is similar to buffer
  • Producers don’t need to care about whether consumers receive messages
• Producers push to broker
• Broker pushes to consumers
• Producers have control of their push rate, consumers have control of their poll rate

Note we can also have a push-poll model - producers push to broker, consumers poll from broker as needed.

• Advantage
  • Broker doesn’t need to worry about bad connection to consumer
    • Can simply ask for data when connection is better
• Disadvantage
  • Frequent polling for infrequently arriving data is wasteful