Google Cloud's London Outage: Testing Your Zone Failover Plan

Most mid-sized IT teams have a cloud resilience story they tell auditors and insurers: “we run across multiple availability zones.” For many, that is the whole plan. The workloads were deployed that way once, the architecture diagram shows three boxes, and nobody has watched one of those boxes go dark.

On October 20, one did. The way it failed is worth studying, because spreading across zones was not enough for everyone.

What happened in London

Google’s incident page puts the start at 23:00 Pacific time on October 19, which was 7:00 a.m. in London on October 20, and the end at 05:10 Pacific (1:10 p.m. in London), six hours and ten minutes later. The primary impact was in one zone, europe-west2-b, of the London region. Dozens of products were listed as affected, including Compute Engine, Google Kubernetes Engine, Cloud SQL and Persistent Disk.

Google’s preliminary report, posted on October 22, describes the cause. A grid disruption moved the data center onto its battery backup (UPS), but the automation that should have started the diesel generators “failed to initiate the logical automatic start/transfer operations of the generators.” When the batteries ran out, the zone went down. Google says a final report with preventive actions will follow, so the details may change.

The effects reached well beyond Google’s own customers. The Register listed Lloyds Bank, HSBC, Royal Mail, DPD and Revolut among the firms reported as affected. Silicon UK reported that the video system used for remote court hearings in England and Wales was disrupted, and that by 5 p.m., hours after Google’s listed end time, Revolut marked only six of its 27 services as working normally.

Why “multi-zone” was not the whole answer

The most useful part of the preliminary report is about the zones that kept their power. Google says some customers running workloads in europe-west2-a and europe-west2-c were also affected, for two reasons:

• Demand surges. When one zone failed, workloads rushed to the others, causing “instantaneous demand surges on unimpacted compute capacity.” If your failover plan assumes you can launch new servers in a healthy zone at the moment you need them, you are competing with every other customer doing the same thing.
• Control-plane dependencies. Google describes “synchronous control-plane validation dependencies across zonal boundaries” that led to API timeouts while those calls waited on the offline zone. Creating, resizing or changing resources can fail even when the servers already running are fine. Google also says multi-zonal and regional services spanning the region saw elevated latency.

This matches long-standing cloud design guidance. Google’s own documentation, updated October 22, says that zones “should be considered a single failure domain within a region” and recommends deploying across multiple zones for high availability, with a separate disaster recovery plan for losing an entire region. AWS calls the related principle static stability: rather than relying on the control plane to launch new instances during a zone failure, “having that extra capacity pre-provisioned helps achieve static stability.” The idea applies to any provider: the most reliable recovery is the one that needs the fewest changes while the failure is happening.

Most of your exposure is probably in SaaS

Few mid-sized companies run large workloads directly in a public cloud region. More often, their exposure is indirect, through the software-as-a-service tools they depend on: payroll, payments, ticketing, ERP, phone systems and line-of-business apps. Revolut’s customers did not choose a Google zone. Their bank did.

That is why the useful question is not only “how is our cloud built?” but “which of our critical services share one region, and what does each vendor do when that region has a bad day?”

A practical review for the next month

You do not need a large engineering team to get real answers. Work through these steps for your top five to ten business-critical systems.

1. Map where each system actually runs. For workloads you host, record the provider, region and zones. For SaaS, ask each vendor which hosting provider and region serve your account and whether it fails over to a second region. Note where several critical systems land in the same place.
2. Separate “running” from “changeable.” For each workload, ask whether it stays up if you cannot create, resize or reconfigure anything for six hours. Autoscaling groups, Kubernetes clusters that add nodes on demand, and recovery scripts that launch new servers all depend on the control plane.
3. Decide where pre-provisioned capacity is worth paying for. For the few systems that must survive a zone loss, consider running enough capacity in each zone to carry the load without launching anything new. Providers sell guaranteed capacity, but it is not free: Google’s reservation documentation says you are charged for reserved resources at the on-demand rate for as long as the reservation exists. This is a business decision about cost against downtime, and it should be made explicitly.
4. Check your databases and storage. A web tier spread across zones does not help if the database sits in one zone. Confirm which data services are zonal and which are regional or replicated, and how long a failover takes in practice.
5. Test a zone loss. Schedule a game day in which you take one zone out of service for a non-production copy of a key system, or for production in a quiet window if your design allows it. Time the recovery and write down what needed a human.
6. Plan for the provider’s “fixed” not being your “fixed.” Revolut was still restoring services hours after Google’s listed end time. Your runbook should cover restarting queues, clearing backlogs and checking data consistency, not just waiting for the status page to turn green.
7. Subscribe to the right status pages. Assign someone to follow your cloud provider’s and key SaaS vendors’ status feeds, and decide in advance who tells the business what.

Questions to ask your vendors and your team

• Which region and hosting provider serve our account, and do you fail over to a second region?
• When did you last test losing a zone or a region, and what was the recovery time?
• Which of our own workloads need the cloud control plane to recover, and which keep running without it?
• Where would a zone failure leave us short of capacity, and what would guaranteed capacity cost?
• Are our databases zonal, regional or replicated across regions?
• How many of our critical systems depend on the same region?
• What does our contract say about service credits, and do they come close to covering our actual losses? Talk to your attorney if the answer matters.
• Who decides to fail over, and who communicates with staff and customers while it happens?

If you would like an independent review of your cloud and SaaS resilience, get in touch.