Final Update: Monday, July 10th 2017 21:07 UTC
We’ve confirmed VSTS is back to a healthy state as have many customers. The AAD team helped us recover from this issue by increasing throttling limits which helped the service drain out a backlog of auth related requests. Please note that we are still reviewing our telemetry to ensure there are no residual effects.
This was a major outage for our service and we’re committed to follow-up with a detailed post mortem that clarifies what happened and how we avoid a similar issue in the future. We’ll try to publish this post mortem before end of week in a new blog post.
Please accept our apologies for the impact this had on you and your team.
Sincerely,
Tom
Update: Monday, July 10th 2017 20:11 UTC
The overall AAD issue has been resolved and their status is currently green in the Azure Service Health Dashboard. The continuing issues in VSTS are related to a high rate of calls our service is making to AAD as it recovers from the incident. We are working with the AAD team to push out a mitigation that we hope will resolve the issue for VSTS. We’ll provide another update soon.
- Next Update: Before Monday, July 10th 2017 21:15 UTC
Sincerely,
Tom
Update: Monday, July 10th 2017 18:53 UTC
The AAD team applied an initial mitigation however we continue to experience issues related to authentication and authorization. The root cause is still not fully understood and we’re coordinating with our AAD partners. Customers will experience issues with login using both AAD and Microsoft account (e.g. – <user>@outlook.com) and intermittent errors when interacting with the service.
- Next Update: Before Monday, July 10th 2017 21:15 UTC
Sincerely,
Tom
Update: Monday, July 10th 2017 17:29 UTC
There is an active issue with Azure Active Directory (AAD) that is impacting some VSTS causing login errors. Additionally, some logged in users may intermittent HTTP 500 errors when interacting with the service for this URL – https://app.vssps.visualstudio.com/_signedin:
The AAD Team is aware of this issue and actively working to mitigate it. An status update is being posted by the AAD Team on the Azure Service Health Dashboard – https://azure.microsoft.com/en-us/status/
We’ll provide an update when we know more or in the next 2 hours.
- Next Update: Before Monday, July 10th 2017 19:45 UTC
Sincerely,
Tom
Initial Update: Monday, July 10th 2017 16:44 UTC
We are actively investigating login issues with Visual Studio Team Services. Customers using VSTS will likely be impacted and experience login failures when trying to access the service. Some VSTS customers may have cached tokens that enable them to login to the service however they may experience other errors in the portal due to the general auth issues.
- Next Update: Before Monday, July 10th 2017 17:45 UTC
Sincerely,
Sri Harsha