Incident: A configuration change inadvertently removed a critical resource required by one of our services. While existing infrastructure continued to operate normally, newly scaled infrastructure could not become ready. As morning traffic increased, there was a growing gap between incoming requests and our system's capacity to handle them, resulting in service degradation. We resolved the issue by manually restoring the missing resource and correcting the configuration change that caused its removal.
Impact: For approximately 1.5 hours, a subset of customers experienced a full outage of our web application and desktop applications. During this time, affected users were unable to access Asana. No customer data was lost.
Moving forward: We are implementing additional safeguards in our configuration management process to prevent critical resources from being inadvertently removed. We are also enhancing our monitoring systems to detect degradation earlier, before it impacts customer access.
Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.