Shortly after a major spike in site visits, the Toucan website (https://jointoucan.com) was down from approximately 12 pm to 4 pm. Over the course of a few hours, and several 3-minute emergency Headspace meditation sessions, our website was fully operational. This outage affected our website and had no impact on the operations of the Toucan Chrome Extension.
Toucan operates on AWS Elastic Beanstalk with Load Balancers to ensure that we can handle increased spikes in traffic. Usually this operates fine, and we are thrilled to see our load balancers coming into play. Unfortunately today, there was a ELB operational issue on AWS in our particular region. This became particularly troublesome when our load balancer wanted to add more instances. It resulted in a "OutOfService" state on our EC2 instance. Now, upon seeing this, the issue was made worse by restarting the environment and trying to re-spin up the load balancers, as that increased the provisioning and registration times of our new load balancers and instances. After working with the incredible support team at AWS, and several additional attempts at registering an instance to the load balancer, it was successful once the underlying issue in our region was fixed.
Patience is a virtue. Truly. Rebuilding our environment made the situation worse. What would have resolved this issue the fastest is taking a step back, and creating a new environment in a different region from a configuration (that we now have).
Users love it when Toucan works. Such a humbling experience to have gone through an event like this where users that love Toucan were right by our side supporting us to get it back up. Genuinely, we are here for our users, but today showed that they want to support us too and that means the world to us.
Team is your greatest strength. Every day I'm so grateful to work with such an incredible team at Toucan, and the way the team handled today only furthers that gratitude. I learn so much from our team on a daily basis and what they did today only further proves their love for our mission, our users, and each other. Knowing that they were winning at life and staying in touch with our users gave me no hesitation to work with AWS to resolve the issues as quickly as possible