Final Update: Monday, October 23rd 2017 14:57 UTC
We’ve confirmed that all systems are back to normal. Some customers experienced web experience failures, slow page loads, and hosted build failures in the Central US Region. User may have also seen poorly formatted web pages and 504 errors in their builds. Sorry for any inconvenience this may have caused.
Root Cause: Our CDN provider has confirmed that a Point of Presence (POP) in Texas experienced network failures. They have shut down the faulty circuits and that has resolved the issue. We're working with them to understand what happened and get a more detailed root cause analysis and prevention plan.
- Incident Timeline: 2 hours & 13 minutes – October 23rd 2017 12:25 UTC through October 23rd 2017 14:38 UTC
Sincerely,
Daniel
Update: Monday, October 23rd 2017 14:01 UTC
We continue to investigate issues with the VSTS web experience and Hosted Build. Users in the Central US region may experience slow web page load times, connection failures, build failures (504 errors), and/or an abnormal web page UI (wrong format).
Root cause is not fully understood at this time, though it is suspected to be problems with our Content Delivery Network (CDN). We've engaged our CDN partners and are working on a plan to resolve the issue. We currently have no estimated time for resolution.
Web experience users may observe the following error message: "Request failed: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond".
Hosted build users may observe 504 gateway timeout errors.
- Work Around: Some users have reported that reloading the page may help work around this issue.
- Next Update: Before Monday, October 23rd 2017 15:15 UTC
Sincerely,
Daniel
Initial Update: Monday, October 23rd 2017 13:19 UTC
We are investigating performance issues affecting users in the Central US Region. Users in may experience degraded performance or slow page load times.
Initial investigation indicates there may be a CDN issue in Central US Region.
- Next Update: Before Monday, October 23rd 2017 14:30 UTC
Sincerely,
Niall