On March 14th, 2023, at roughly 16:32 UTC, we modified an SSL configuration which caused our ingestion endpoint (events.hookdeck.com) to incorrectly return a 301 status code. This lasted for about 2 minutes, at which point we reverted the configuration. However during that time we failed to ingest webhooks sent to our platform. For most, if the calling party followed redirects, hookdeck will have responded with a 405, the result of following the redirect with a GET.

While we expect that most providers’ retry logic will have reissued the request, we did fall short today on our mission to not drop a webhook for the first time in a year and a half. We are putting a series of measures in place to mitigate this for the future.

Timeline

• 16:31 UTC: Implemented SSL configuration change
• 16:33 UTC: Reverted configuration change and restored proper ingestion

Lessons learned

What went well

• The inability to ingest did not last long as we caught the mistake quickly.
• Since we replied with a non 2xx error code, providers likely retried the failed request in most cases.

What went wrong

• The fact that our ingestion endpoint (events.hookdeck.com) is on the same domain as our other services (api.hookdeck.com, dashboard.hookdeck.com, …), exposes us to accidental and unintended configuration changes.

Corrective actions

• Our systems for ingestion and for events processing are decoupled. In order to fully separate them, we are working to put in place a new domain (hkdk.events) for event ingestion which will help us avoid configuration changes inadvertently affecting other systems.