Final Update: Thursday, August 31st 2017 17:15 UTC
VSTS in the South Central region is now back in a healthy state. The incident was mitigated after the bad web tier node was taken out of the load balancer. We've confirmed that users are no longer seeing slowness or errors. Additionally, several users have confirmed builds are now completing without delays.
We are working to analyze the memory dump from the bad web node to understand root cause so we can be sure the issue doesn't happen again.
Sincerely,
Tom
Update: Thursday, August 31st 2017 16:29 UTC
We are experiencing intermittent issues in VSTS in the South Central region. Users will see symptoms including slowness and failures (e.g. - HTTP 503) using the web experience and some builds not completing. This root cause appears to be related to a single web tier node that started after a recent deployment. We've taken a memory dump on this node and pulled it out of the network load balancer. We're now working to confirm if this resolved the issue and will follow-up soon when we know more.
- Next Update: Before Thursday, August 31st 2017 17:45 UTC
Sincerely,
Tom
Initial Update: Thursday, August 31st 2017 15:47 UTC
A potentially customer impacting alert is being investigated. Triage is in progress and we will provide an update with more information.
- Next Update: Before Thursday, August 31st 2017 17:00 UTC
Sincerely,
Niall