Google Cloud June 2025: quota-policy control-plane change crashed API admission globally

Original: Google Cloud Service Health, detailed incident report dated 13 June 2025. Distinguish the 10:51 PDT start and broad API restoration by roughly 13:49 PDT from the later 18:18 PDT final recovery of Vertex AI Online Prediction; the status page’s header runs 10:51–18:18 PDT. Use these different endpoints explicitly, not an undifferentiated seven-hour outage for every product.

Google’s Service Control checks API authorization, quota and policy using regionally deployed binaries/datastores with near-immediate global metadata replication. A binary released on 29 May contained an unexercised null-pointer path. A 12 June policy insertion included blank fields, replicated across regions within seconds, and sent binaries into crash loops. Engineers identified root cause in ten minutes, disabled the path with a red-button and then encountered restart load in us-central1; external API requests returned elevated 503s. Google listed Vertex Gemini API, Agent Assist, Contact Center AI and other AI services among affected products; some Model Garden models had lingering Vertex AI Online Prediction 5xx responses until later on 12 June. No model weight, training compute or malicious agent was implicated.

Google says its own Cloud Service Health updates were delayed about an hour because the status infrastructure shared the failure, while some customers’ on-cloud monitoring also failed. Proposed mitigations: fail-open on isolated nonessential checks, stage globally replicated data, feature-flag binary changes, protect retry paths with randomized exponential backoff, and improve out-of-band incident communications. Security trade-off: failing open for quota checks is not a blanket recommendation to skip identity or safety authorization. Paired with Cloudflare’s contemporaneous reported third-party storage outage, these records suggest a possible common supplier, but Cloudflare’s original postmortem does not name the supplier, so the cross-provider named link is not proven by these two originals alone. See systemic failure synthesis.