We are going to cover this in 5 simple steps.

  1. Build
  2. Tune
  3. Monitor
  4. Resile
  5. SOS

Step 1: Build

Before we start building there are some things we have to learn so let's start with that.

WHY DISTRIBUTED TASK QUEUES

"people do not like to wait" or do we?

Let's see an example let's assume I am running a mall and I want to keep track of how many people come into the mall. So I did what was the easiest I asked one person to take his laptop and sit at the gate and click a button on my web app which increments the count in my DB and at the end of the day I will just see the DB value.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/8917bfff-f831-48ff-8493-955597205be2/Copy_of_Untitled_Diagram.jpg

The locality in which my mall was located in was very fond of football so I thought I will screen all FIFA World Cup matches in the mall to increase the popularity and it also worked we got a huge footfall in the mall and I was very excited to see the numbers in my DB. But when I checked the number was significantly low. So I went to the person-in-charge of clicking the button and I ask them why the number is so low. So he replied the footfall was very high and he kept pressing the button but it wasn't responding as fast as the people were coming in.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/7dc85681-b681-4992-81c3-983b8f34a520/cat.gif

Which made sense because there is a limit for the server to handle HTTP requests and each click makes an HTTP request which goes to the webserver then updates the DB and then the request cycle completes.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/dd728922-36d9-458a-b732-3ba63e2427c5/Untitled_Diagram_(25).jpg

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/ddfb0287-9c2d-40f8-af77-e479f00f4e54/obama.gif