Delays in commit processing

Timeline

1. 09 Sep · 19:29 UTC 🟢 Resolved atom

   Sep 1, 16:01 UTC
   Resolved - On September 1, 2026, between approximately 14:01 and 16:01 UTC, updates in response to pushes were delayed, temporarily showing stale diffs. The median time to refresh a diff after a push rose from the normal level of about 3 seconds to over 2 minutes at the peak, and more than 140,000 customer accounts had at least one delayed refresh during the most affected 75 minutes. Pushing commits and opening pull requests continued to work normally. The incident was caused by a sharp, concentrated surge in push volume that saturated worker pools and job queueing infrastructure. Autoscaling did not increase capacity as intended, so the backlog did not clear on its own.
   
   The incident was mitigated by manually scaling the affected worker pools and increasing push-processing capacity. This allowed the system to process the backlog, after which refresh times returned to normal. To reduce the likelihood and impact of similar incidents, we are adding quotas and throttling earlier in the push path so a single concentrated source of load cannot saturate shared capacity, improving worker-pool autoscaling so capacity is added automatically, and improving monitors for background job processing so on-call is paged before customers experience delayed pull request updates.
   
   Sep 1, 16:00 UTC
   Update - Time to update pull request diffs have improved to normal thresholds.
   
   Sep 1, 15:00 UTC
   Update - Diffs in the PR view may be stale for several minutes. We are investigating and scaling up resources.
   
   Sep 1, 15:00 UTC
   Investigating - We are investigating reports of degraded performance for Pull Requests

2. 01 Sep · 16:01 UTC 🟡 Resolved atom

   Sep 1, 16:01 UTC
   Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
   
   Sep 1, 16:00 UTC
   Update - Time to update pull request diffs have improved to normal thresholds.
   
   Sep 1, 15:00 UTC
   Update - Diffs in the PR view may be stale for several minutes. We are investigating and scaling up resources.
   
   Sep 1, 15:00 UTC
   Investigating - We are investigating reports of degraded performance for Pull Requests

3. 01 Sep · 16:00 UTC 🟡 Identified atom

   Sep 1, 16:00 UTC
   Update - Time to update pull request diffs have improved to normal thresholds.
   
   Sep 1, 15:00 UTC
   Update - Diffs in the PR view may be stale for several minutes. We are investigating and scaling up resources.
   
   Sep 1, 15:00 UTC
   Investigating - We are investigating reports of degraded performance for Pull Requests

4. 01 Sep · 15:00 UTC 🟡 Identified atom

   Sep 1, 15:00 UTC
   Update - Diffs in the PR view may be stale for several minutes. We are investigating and scaling up resources.
   
   Sep 1, 15:00 UTC
   Investigating - We are investigating reports of degraded performance for Pull Requests