Final Update: Saturday, March 24th 2018 13:09 UTC
We’ve confirmed that all systems are back to normal as of 11:44 UTC. Our logs show the incident started on 2018/03/24 23:46 UTC and that during the 12 hours and 02 minutes that it took to resolve the issue. During the impact time, customer experience delay in emails notifications. Sorry for any inconvenience this may have caused.
- Root Cause: There is a special character in the email header causing email job to fail.
- Chance of Re-occurrence: Medium pending a hotfix deployment.
- Incident Timeline: 12 hours and 02 minutes – 2018/03/23 11:44 UTC through 2018/03/24 23:46 UTC
Sincerely,
Anmol Bhasin
Update: Saturday, March 24th 2018 07:43 UTC
Our DevOps team identified the issue causing this email notification failures. and applied a quick mitigation. Currently back ground job is clearing the backlog email queue and it is expected to take few hour for all backlog to finish. Root cause is fully understood at this time. There will be a hotfix deployed on Monday to avoid reoccurrence of this issue, The problem began at 2018-03-21 06:00 AM UTC. We have the necessary detection added for the weekend to get an alert if issue reoccur.
- Next Update: Before Saturday, March 24th 2018 19:45 UTC
Sincerely,
Anmol Bhasin
Update: Saturday, March 24th 2018 01:58 UTC
Our DevOps team identified the issue causing this email notification failures. and applied a quick mitigation. Currently back ground job is clearing the backlog email queue and it is expected to take few hour for all backlog to finish. Root cause is fully understood at this time. There will be a hotfix deployed on Monday to avoid reoccurrence of this issue, The problem began at 2018-03-21 06:00 AM UTC. We will have necessary detection added for the weekend to get an alert if issue reoccur.
- Next Update: Before Saturday, March 24th 2018 06:00 UTC
Sincerely,
Sudheer Kumar
Initial Update: Saturday, March 24th 2018 00:23 UTC
We're investigating issues with emails not been sent for any account creation, account invitation, changes to credentials and order confirmation for marketplace
- Next Update: Before Saturday, March 24th 2018 03:00 UTC
Sincerely,
Sudheer Kumar