Final Update: Tuesday, June 12th 2018 17:43 UTC
We’ve confirmed that all systems are back to normal as of June 12th 17:15 UTC. Our logs show the incident started on June 12th 13:05 UTC and that during the 4 hours and 15 minutes that it took to resolve the issue, customers in West Europe who queued requests against the VS2017 Hosted Agent Pools experienced slow aquisition times. Sorry for any inconvenience this may have caused.
- Root Cause: The failure was due to an addition to a stored procedure in our database which caused an unexpected increase in the amount of blocking, causing request assignments to be delayed.
- Chance of Re-occurrence: Low - Hotfix is currently rolling out to the remaining regions
- Lessons Learned: We further investigating how to speed up recovery of the Hosted Agent pools for other incidents.
- Incident Timeline: 4 hours & 15 minutes – June 12th 13:05 UTC through June 12th 17:15 UTC
Sincerely,
Dexter
Update: Tuesday, June 12th 2018 15:51 UTC
Our DevOps team continues to investigate issues with VS2017 Hosted Agent delays in West Europe. Root cause is not fully understood at this time, though it is suspected to be a regression with a recent deployment. We have patched what is suspected to be the underlying issue, and are upgrading out database to hopefully mitigate the issue. The problem began at June 12th 13:00 UTC. We currently have no estimated time for resolution.
- Work Around: If possible, target other Hosted Agent pools
- Next Update: Before Tuesday, June 12th 2018 18:00 UTC
Sincerely,
Dexter
Initial Update: Tuesday, June 12th 2018 14:25 UTC
- We're investigating delays in assigning Hosted Agents in West Europe.
- Users in West Europe may see increased delays in getting hosted agents assigned to their request
- This seems to be due to a regression in a recent deployment
- We are working on mitigating the incident
Next Update: Before Tuesday, June 12th 2018 15:00 UTC
Sincerely,
Dexter