Incident: During a routine hardware change, a misconfiguration of our load balancer health checks caused EU load balancers to be marked as unhealthy, rejecting all requests from users.
Impact: Asana was unavailable to load new sessions for EU users for 135 minutes, with only existing sessions able to be used. No customer data was lost.
Moving forward: As a result of this incident, we are working to simplify load balancer architecture and improve operational processes around hardware changes.
Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.