📋 Summary

On October 14, 2025, at 21:00 UTC, a migration on the preferences collection was initiated to clean up duplicate records and update collection indexes.

Although the migration itself completed successfully, subsequent index maintenance operations triggered a severe performance degradation across MongoDB shards, resulting in high CPU utilization, increased response times (up to 1 minute for certain endpoints).

The issue recurred on October 15 during additional index maintenance, resulting in a temporary downtime for Inbox-related endpoints. After extensive investigation with the MongoDB Support team, the problem was traced to inefficient index planning on MongoDB that caused excessive load on primary and secondary nodes.

Mitigation steps were rolled out progressively, culminating in a final fix to the preferences query around 20:05 UTC, which restored normal API performance.

💥 Impact

The US region was only affected. The EU region was performing normally.

• Duration:
  • First incident: ~21:15 UTC (Oct 14) – 01:50 UTC (Oct 15)
  • Second incident: ~10:00 UTC (Oct 15) – 20:05 UTC (Oct 15)
• Services affected:
  • API endpoints related to Inbox and Subscribers Preferences
  • Worker Queues and jobs execution
  • Elevated latency across all API services during peak degradation
  • Temporary downtime for Inbox endpoints between 10:40 and 11:05 UTC on October 15
• User impact:
  • API consumers saw timeouts and slow responses on affected endpoints
  • End users experienced delayed notification deliveries
  • No data loss occurred

🧾 Timeline

Legend:

🟠 - API response times degraded