Final Update: Friday, December 1st 2017 20:16 UTC
We’ve confirmed that all systems are back to normal as of Friday, December 1st 2017 19:20 UTC. Our logs show the incident started on Friday, December 1st 2017 18:50 UTC. and that during the 30 minutes that it took to resolve the issue. 1.6% of customers experienced slowness in the South Central region. Sorry for any inconvenience this may have caused.
- Root Cause: The failure was due to a query plan caching issue in one of our databases.
- Chance of Re-occurrence: Low - Well-understood issue with prevention steps
- Lessons Learned: We are working on upgrading these databases to prevent this issue from happening, and evaluating the database sizing in the meantime to minimize the impact when it does happen.
- Incident Timeline: 30 minutes – Friday, December 1st 2017 18:50 UTC through Friday, December 1st 2017 19:20 UTC
Sincerely,
Dexter
Initial Update: Friday, December 1st 2017 19:27 UTC
We're investigating Performance Degradation in South Central US. User in this region may experience slow response on VSTS.
- Next Update: Before Friday, December 1st 2017 20:00 UTC
Sincerely,
Bapayya