Final Update: Saturday, June 23rd 2018 14:22 UTC
We’ve confirmed that all systems are back to normal as of Saturday, June 23rd 2018 14:00 UTC . Our logs show the incident started on Saturday, June 23rd 2018 03:28 UTC and that during the 10 hours and 32 minutes that it took to resolve the issue customers may have experienced delays in Notification and Build in the South Central US Region. Sorry for any inconvenience this may have caused.
- Root Cause: A hotfix was applied to a poorly performing Stored Procedure.
- Chance of Re-occurrence: Low
Sincerely,
Niall
Update: Saturday, June 23rd 2018 13:08 UTC
Our DevOps team continues to investigate issues with Notification and Build delays. Root cause is not fully understood at this time. The problem began at 03:28 UTC 23rd June. We currently have no estimated time for resolution.
- Next Update: Before Saturday, June 23rd 2018 15:15 UTC
Sincerely,
Niall
Initial Update: Saturday, June 23rd 2018 12:01 UTC
We're investigating Performance Degradation in South Central US.
Customers may notice delays in Notifications and Builds during this time.
- Next Update: Before Saturday, June 23rd 2018 13:01 UTC
Sincerely,
Niall