Actions delays in starting runs

Timeline

1. 25 Aug · 01:36 UTC 🟡 Resolved atom

   Aug 24, 14:34 UTC
   Resolved - On August 24, 2026, between 13:33 UTC and 14:04 UTC, 3.8% of Actions runs experienced start delays over 5 minutes with 1.25% of Actions runs failing outright.
   
   The incident was caused by a disk failure on a node hosting one of many service instances responsible for processing runner assignment events. Typically, pods on unhealthy nodes are removed and replaced automatically without impact. In this case, although the node was severely degraded and unable to perform disk operations, it continued sending healthy signals, preventing the system from immediately moving its work elsewhere. During this period, events assigned to the affected component accumulated until an automatic rebalance redirected processing to healthy components at 13:54 UTC. The queue backlog was cleared at 14:00 UTC, and processing returned to normal by 14:04 UTC.
   
   To prevent a recurrence, we are improving detection and automated remediation for unhealthy nodes that aren’t fully offline. We are also strengthening application-level resiliency, so stalled consumers are automatically removed quickly and their work reassigned without waiting for the affected node to recover.
   
   Aug 24, 14:26 UTC
   Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
   
   Aug 24, 14:22 UTC
   Update - Failures while queuing and running Actions jobs for a subset of customers are now resolving. We are monitoring for full recovery.
   
   Aug 24, 13:56 UTC
   Investigating - We are investigating reports of degraded performance for Actions

2. 24 Aug · 14:34 UTC 🟠 Resolved atom

   Aug 24, 14:34 UTC
   Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
   
   Aug 24, 14:26 UTC
   Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
   
   Aug 24, 14:22 UTC
   Update - Failures while queuing and running Actions jobs for a subset of customers are now resolving. We are monitoring for full recovery.
   
   Aug 24, 13:56 UTC
   Investigating - We are investigating reports of degraded performance for Actions

3. 24 Aug · 14:34 UTC 🟠 Resolved atom

   Aug 24, 14:34 UTC
   Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
   
   Aug 24, 14:26 UTC
   Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
   
   Aug 24, 14:22 UTC
   Update - Failures while queuing and running Actions jobs for a subset of customers are now resolving. We are monitoring for full recovery.
   
   Aug 24, 13:56 UTC
   Investigating - We are investigating reports of degraded performance for Actions

4. 24 Aug · 14:26 UTC 🟠 Monitoring atom

   Aug 24, 14:26 UTC
   Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
   
   Aug 24, 14:22 UTC
   Update - Failures while queuing and running Actions jobs for a subset of customers are now resolving. We are monitoring for full recovery.
   
   Aug 24, 13:56 UTC
   Investigating - We are investigating reports of degraded performance for Actions

5. 24 Aug · 13:56 UTC 🟡 Investigating atom

   Aug 24, 13:56 UTC
   Investigating - We are investigating reports of degraded performance for Actions