Incident: A server version change resulted in some servers in the European and Australian data centers becoming incapable of serving new sessions, leading to users being unable to load Asana.
Impact: From 3:19 UTC to 8:36 UTC, a significant portion of attempts to start new sessions failed for customers with data residency in Europe or Australia. ~20% of requests failed in Europe and ~80% failed in Australia. Existing sessions were unaffected, and there was no loss of customer data.
Moving forward: We have added monitoring to alert us of this failure case, and have also added safeguards to the deployment process to prevent servers from reaching this state again.
Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.