Incident with Actions

Timeline

1. 05 Nov · 00:04 UTC 🟡 Resolved atom

   Oct 22, 15:18 UTC
   Resolved - On October 22, 2026, from 13:05 to 15:20 UTC, our event streaming platform was degraded resulting in elevated job processing latency spikes of 19-35 seconds and elevated job processing times of 4m – this resulted in delayed Actions workflow and step results (up to 13m), notifications (20m) and repository push events (12m). Recovery for repositories began at 14:32, with Actions recovery beginning at 14:59. The issue reoccurred at 16:26 again impacting Actions runs, repo push events, and notifications. Recovery began at 17:16 and completed at 2:07.
   
   The incident was caused by a significant increase in load coupled with client connection storms during a deployment overloading our Kafka event streaming brokers.
   
   To prevent future recurrence, we have expanded the affected cluster, shifted traffic to less loaded clusters, and implemented tunable client reconnect backoff.
   
   Oct 22, 15:18 UTC
   Update - GitHub Actions workflow and step result processing has recovered. Repository update processing has also returned to normal.
   
   Oct 22, 15:17 UTC
   Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
   
   Oct 22, 15:06 UTC
   Update - We’re seeing recovery, and repository update processing has returned to normal. Delays affecting GitHub Actions workflow and step results are continuing to clear as we scale processing capacity; the remaining backlog is expected to take about 20 minutes to drain.
   
   Oct 22, 14:35 UTC
   Update - Repository update processing has recovered. We’re continuing to work on delays affecting GitHub Actions workflow and step results. We’ll share another update as we make progress.
   
   Oct 22, 14:19 UTC
   Update - We’re investigating delays in processing GitHub Actions workflow and step results, along with some repository updates.
   
   Oct 22, 14:18 UTC
   Investigating - We are investigating reports of degraded performance for Actions

2. 29 Oct · 12:25 UTC 🟡 Resolved atom

   Oct 22, 15:18 UTC
   Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
   
   Oct 22, 15:18 UTC
   Update - GitHub Actions workflow and step result processing has recovered. Repository update processing has also returned to normal.
   
   Oct 22, 15:17 UTC
   Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
   
   Oct 22, 15:06 UTC
   Update - We’re seeing recovery, and repository update processing has returned to normal. Delays affecting GitHub Actions workflow and step results are continuing to clear as we scale processing capacity; the remaining backlog is expected to take about 20 minutes to drain.
   
   Oct 22, 14:35 UTC
   Update - Repository update processing has recovered. We’re continuing to work on delays affecting GitHub Actions workflow and step results. We’ll share another update as we make progress.
   
   Oct 22, 14:19 UTC
   Update - We’re investigating delays in processing GitHub Actions workflow and step results, along with some repository updates.
   
   Oct 22, 14:18 UTC
   Investigating - We are investigating reports of degraded performance for Actions

3. 22 Oct · 15:18 UTC 🟡 Resolved atom

   Oct 22, 15:18 UTC
   Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
   
   Oct 22, 15:18 UTC
   Update - GitHub Actions workflow and step result processing has recovered. Repository update processing has also returned to normal.
   
   Oct 22, 15:17 UTC
   Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
   
   Oct 22, 15:06 UTC
   Update - We’re seeing recovery, and repository update processing has returned to normal. Delays affecting GitHub Actions workflow and step results are continuing to clear as we scale processing capacity; the remaining backlog is expected to take about 20 minutes to drain.
   
   Oct 22, 14:35 UTC
   Update - Repository update processing has recovered. We’re continuing to work on delays affecting GitHub Actions workflow and step results. We’ll share another update as we make progress.
   
   Oct 22, 14:19 UTC
   Update - We’re investigating delays in processing GitHub Actions workflow and step results, along with some repository updates.
   
   Oct 22, 14:18 UTC
   Investigating - We are investigating reports of degraded performance for Actions

4. 22 Oct · 15:06 UTC 🟡 Identified atom

   Oct 22, 15:06 UTC
   Update - We’re seeing recovery, and repository update processing has returned to normal. Delays affecting GitHub Actions workflow and step results are continuing to clear as we scale processing capacity; the remaining backlog is expected to take about 20 minutes to drain.
   
   Oct 22, 14:35 UTC
   Update - Repository update processing has recovered. We’re continuing to work on delays affecting GitHub Actions workflow and step results. We’ll share another update as we make progress.
   
   Oct 22, 14:19 UTC
   Update - We’re investigating delays in processing GitHub Actions workflow and step results, along with some repository updates.
   
   Oct 22, 14:18 UTC
   Investigating - We are investigating reports of degraded performance for Actions

5. 22 Oct · 14:35 UTC 🟡 Identified atom

   Oct 22, 14:35 UTC
   Update - Repository update processing has recovered. We’re continuing to work on delays affecting GitHub Actions workflow and step results. We’ll share another update as we make progress.
   
   Oct 22, 14:19 UTC
   Update - We’re investigating delays in processing GitHub Actions workflow and step results, along with some repository updates.
   
   Oct 22, 14:18 UTC
   Investigating - We are investigating reports of degraded performance for Actions

6. 22 Oct · 14:19 UTC 🟡 Identified atom

   Oct 22, 14:19 UTC
   Update - We’re investigating delays in processing GitHub Actions workflow and step results, along with some repository updates.
   
   Oct 22, 14:18 UTC
   Investigating - We are investigating reports of degraded performance for Actions