Disruption with Dependabot Alerts

Timeline

1. 26 Oct · 16:37 UTC 🟠 Resolved atom

   Oct 23, 18:51 UTC
   Resolved - We identified that an advisory-publication driven fan-out was overwhelming Dependabot alert processing on repositories by saturating the queue with its own events, delaying processing of events related to repository pushes.
   
   We have now isolated this fan-out and applied a throttle so it runs at a fixed rate and cannot delay other events. This remediation has successfully prevented a new fan-out process impacting users.
   
   Oct 23, 18:36 UTC
   Monitoring - The degradation has been mitigated. We are monitoring to ensure stability.
   
   Oct 23, 17:27 UTC
   Update - Some Dependabot alerts continue to be delayed in closing after repository updates address a vulnerability.
   
   We're continuing to investigate and work to restore normal processing.
   
   Oct 23, 16:43 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring progress and will share another update as we learn more.
   
   Oct 23, 16:11 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring the rollout and will share another update as we learn more.
   
   Oct 23, 15:39 UTC
   Update - We’re scaling alert processing capacity to reduce the backlog causing delays in some Dependabot alerts closing after affected repositories have been updated. We’re continuing to monitor and will share another update as more information becomes available.
   
   Oct 23, 15:00 UTC
   Update - We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.
   
   Oct 23, 14:58 UTC

2. 23 Oct · 18:51 UTC 🟠 Resolved atom

   Oct 23, 18:51 UTC
   Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
   
   Oct 23, 18:36 UTC
   Monitoring - The degradation has been mitigated. We are monitoring to ensure stability.
   
   Oct 23, 17:27 UTC
   Update - Some Dependabot alerts continue to be delayed in closing after repository updates address a vulnerability.
   
   We're continuing to investigate and work to restore normal processing.
   
   Oct 23, 16:43 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring progress and will share another update as we learn more.
   
   Oct 23, 16:11 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring the rollout and will share another update as we learn more.
   
   Oct 23, 15:39 UTC
   Update - We’re scaling alert processing capacity to reduce the backlog causing delays in some Dependabot alerts closing after affected repositories have been updated. We’re continuing to monitor and will share another update as more information becomes available.
   
   Oct 23, 15:00 UTC
   Update - We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.
   
   Oct 23, 14:58 UTC
   Investigating - We are investigating reports of impacted performance for some GitHub services.

3. 23 Oct · 18:36 UTC 🟠 Monitoring atom

   Oct 23, 18:36 UTC
   Monitoring - The degradation has been mitigated. We are monitoring to ensure stability.
   
   Oct 23, 17:27 UTC
   Update - Some Dependabot alerts continue to be delayed in closing after repository updates address a vulnerability.
   
   We're continuing to investigate and work to restore normal processing.
   
   Oct 23, 16:43 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring progress and will share another update as we learn more.
   
   Oct 23, 16:11 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring the rollout and will share another update as we learn more.
   
   Oct 23, 15:39 UTC
   Update - We’re scaling alert processing capacity to reduce the backlog causing delays in some Dependabot alerts closing after affected repositories have been updated. We’re continuing to monitor and will share another update as more information becomes available.
   
   Oct 23, 15:00 UTC
   Update - We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.
   
   Oct 23, 14:58 UTC
   Investigating - We are investigating reports of impacted performance for some GitHub services.

4. 23 Oct · 17:27 UTC 🟠 Identified atom

   Oct 23, 17:27 UTC
   Update - Some Dependabot alerts continue to be delayed in closing after repository updates address a vulnerability.
   
   We're continuing to investigate and work to restore normal processing.
   
   Oct 23, 16:43 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring progress and will share another update as we learn more.
   
   Oct 23, 16:11 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring the rollout and will share another update as we learn more.
   
   Oct 23, 15:39 UTC
   Update - We’re scaling alert processing capacity to reduce the backlog causing delays in some Dependabot alerts closing after affected repositories have been updated. We’re continuing to monitor and will share another update as more information becomes available.
   
   Oct 23, 15:00 UTC
   Update - We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.
   
   Oct 23, 14:58 UTC
   Investigating - We are investigating reports of impacted performance for some GitHub services.

5. 23 Oct · 16:43 UTC 🟠 Identified atom

   Oct 23, 16:43 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring progress and will share another update as we learn more.
   
   Oct 23, 16:11 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring the rollout and will share another update as we learn more.
   
   Oct 23, 15:39 UTC
   Update - We’re scaling alert processing capacity to reduce the backlog causing delays in some Dependabot alerts closing after affected repositories have been updated. We’re continuing to monitor and will share another update as more information becomes available.
   
   Oct 23, 15:00 UTC
   Update - We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.
   
   Oct 23, 14:58 UTC
   Investigating - We are investigating reports of impacted performance for some GitHub services.

6. 23 Oct · 16:11 UTC 🟠 Identified atom

   Oct 23, 16:11 UTC
   Update - We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring the rollout and will share another update as we learn more.
   
   Oct 23, 15:39 UTC
   Update - We’re scaling alert processing capacity to reduce the backlog causing delays in some Dependabot alerts closing after affected repositories have been updated. We’re continuing to monitor and will share another update as more information becomes available.
   
   Oct 23, 15:00 UTC
   Update - We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.
   
   Oct 23, 14:58 UTC
   Investigating - We are investigating reports of impacted performance for some GitHub services.

7. 23 Oct · 15:39 UTC 🟠 Identified atom

   Oct 23, 15:39 UTC
   Update - We’re scaling alert processing capacity to reduce the backlog causing delays in some Dependabot alerts closing after affected repositories have been updated. We’re continuing to monitor and will share another update as more information becomes available.
   
   Oct 23, 15:00 UTC
   Update - We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.
   
   Oct 23, 14:58 UTC
   Investigating - We are investigating reports of impacted performance for some GitHub services.

8. 23 Oct · 15:00 UTC 🟠 Identified atom

   Oct 23, 15:00 UTC
   Update - We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.
   
   Oct 23, 14:58 UTC
   Investigating - We are investigating reports of impacted performance for some GitHub services.