Summary

From approximately 2:06pm PDT to 2:27pm PDT on 04/02/25, Opal web pages displayed an error and lacked styling for non on-prem customers

Impact

Opal users on production environments were nondeterministically unable to access several pages on Opal web including Inventory, Catalog, UARs, and requests. Non web surfaces (CLI, Slack, API) were unaffected and available.

Severity: Sev 1

This incident was classified by a SEV 1 in accordance with our SEV guidelines. While not all users were affected, those that were were unable to accomplish necessary tasks in Opal, and blocked for the extent of outage.

Root cause analysis

The root cause was an issue with our color scheme manager that introduced a race condition leading to a syntax error

Actions taken

• Removed extraneous logic that introduced the issue

Timeline

• [April 2, 2:06 PT] - Customers started reporting that Opal was down for them. All sharing screenshots with a similar look that their color scheme was missing
• [April 2, 2:27 PT] - Rolled back all prod instances from 1.951 to 1.950
• [April 2, 2:39] - Got confirmation Opal is up and running again from customers
• [April 2, 3:54 PT] - Opal Engineers found the issue with how we manage color schemes and implemented a fix to be included in 1.952

Next steps

N/A

Conclusion

Opal recognizes that this was a widespread disruptive event for a multitude of customers. We are fully committed to learning from this experience and making improvements to our testing and development cycle.

It is our top priority to ensure our systems are safe, stable and reliable for all our customers. We look forward to working with you and as always, are open to any and all feedback you may have.