Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

Timeline

1. 09 Mar · 05:25 UTC 🟡 Resolved api

   # Incident Report
   ## Summary
   On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, Vertex Gemini API and Dialogflow CX in US regions and the global endpoint. The issue persisted for a duration of 1 hour and 58 minutes.
   This is not the level of quality and reliability we strive to offer you, and we have taken immediate steps to improve the platform’s performance and availability.
   ## Root Cause
   This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. For some specific requests, this created code paths that eventually led to service disruptions and capacity loss for the safety filtering service. Consequently, customers encountered overload (429 and 503) errors for their queries, with some users reporting elevated error rates for specific models in US regions.
   ## Remediation and Prevention
   Google engineers were alerted to the issue via our automated monitoring system on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation.
   Engineers identified the faulty configuration change and initiated a rollback to restore the previous stable configuration. Engineers also added more capacity to the service to stabilize it. Full service restoration was confirmed by 06:35 US/Pacific as the rollback propagated and servers became healthy. \ \
   Google is committed to preventing a repeat of this issue and is taking the following actions:
   * Reinforcing rollout processes to include mandatory validation checkpoints.
   * Improving alerting systems to monitor critical dependencies more closely.
   ## ## Detailed Description of Impact
   On Friday, 27 February 2026 between 04:37 and 06:35 US/Pacific, customers accessing Vertex Gemini APIs may have experienced the following:
   * **Affected Models:** All Vertex AI Gemini API models were affected, including gemini-2.5-flash, gemini-2.5-fl

2. 09 Mar · 05:25 UTC 🟡 Resolved api

   # Incident Report
   ## Summary
   On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, Vertex Gemini API and Dialogflow CX in US regions and the global endpoint. The issue persisted for a duration of 1 hour and 58 minutes.
   This is not the level of quality and reliability we strive to offer you, and we have taken immediate steps to improve the platform’s performance and availability.
   ## Root Cause
   This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. For some specific requests, this created code paths that eventually led to service disruptions and capacity loss for the safety filtering service. Consequently, customers encountered overload (429 and 503) errors for their queries, with some users reporting elevated error rates for specific models in US regions.
   ## Remediation and Prevention
   Google engineers were alerted to the issue via our automated monitoring system on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation.
   Engineers identified the faulty configuration change and initiated a rollback to restore the previous stable configuration. Engineers also added more capacity to the service to stabilize it. Full service restoration was confirmed by 06:35 US/Pacific as the rollback propagated and servers became healthy. \ \
   Google is committed to preventing a repeat of this issue and is taking the following actions:
   * Reinforcing rollout processes to include mandatory validation checkpoints.
   * Improving alerting systems to monitor critical dependencies more closely.
   ## ## Detailed Description of Impact
   On Friday, 27 February 2026 between 04:37 and 06:35 US/Pacific, customers accessing Vertex Gemini APIs may have experienced the following:
   * **Affected Models:** All Vertex AI Gemini API models were affected, including gemini-2.5-flash, gemini-2.5-fl

3. 09 Mar · 05:25 UTC 🟡 Resolved api

   # Incident Report
   ## Summary
   On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates. Impacted services included Google Cloud Support, Agent Assist, Vertex Gemini API and Dialogflow CX in US regions and the global endpoint. The issue persisted for a duration of 1 hour and 58 minutes.
   This is not the level of quality and reliability we strive to offer you, and we have taken immediate steps to improve the platform’s performance and availability.
   ## Root Cause
   This incident was caused by a configuration change to a safety filtering service that supports all Gemini models. For some specific requests, this created code paths that eventually led to service disruptions and capacity loss for the safety filtering service. Consequently, customers encountered overload (429 and 503) errors for their queries, with some users reporting elevated error rates for specific models in US regions.
   ## Remediation and Prevention
   Google engineers were alerted to the issue via our automated monitoring system on Friday, 27 February 2026 04:54 US/Pacific and immediately started an investigation.
   Engineers identified the faulty configuration change and initiated a rollback to restore the previous stable configuration. Engineers also added more capacity to the service to stabilize it. Full service restoration was confirmed by 06:35 US/Pacific as the rollback propagated and servers became healthy. \ \
   Google is committed to preventing a repeat of this issue and is taking the following actions:
   * Reinforcing rollout processes to include mandatory validation checkpoints.
   * Improving alerting systems to monitor critical dependencies more closely.
   ## ## Detailed Description of Impact
   On Friday, 27 February 2026 between 04:37 and 06:35 US/Pacific, customers accessing Vertex Gemini APIs may have experienced the following:
   * **Affected Models:** All Vertex AI Gemini API models were affected, including gemini-2.5-flash, gemini-2.5-fl