Executive Summary
The europe-west2-b report is the clearest public example so far of redundancy failing at the sensor rather than at the supply. A 2N facility with two PLCs, a central generator bank and roughly five minutes of battery still ran six hours and eight minutes of zone outage, and the remedy was a wiring and logic change rather than new equipment. Feed B was open for repairs, so the site was running on one feed when the other one faulted.
Three findings apply to any critical facility. Identical control logic on shared upstream sensors is one failure domain wearing two labels. A single-feed maintenance posture invalidates the automatic transfer path without invalidating the load. And a generator load test proves the plant starts, not that the logic ever asks it to. Google is now validating PLC configurations at sibling sites with similar topologies, which says the pattern is likely to exist elsewhere.
Google published the incident report for its 19 October europe-west2-b outage on 4 November, and it does not blame the generators, the batteries or the switchgear. It blames the logic that decides whether to use them.
Here is the architecture. The site hosting europe-west2-b ran a 2N power design, meaning duplicate paths for every critical load, with a central high-voltage generator bank controlled by two PLCs. Power arrived on two independent feeders, A and B. Batteries carried the load for roughly five minutes while the PLC watched for undervoltage, started the diesels, synchronized them and transferred the site.
Five days before the outage, a quarterly generator load test turned up problems. Feed B’s breaker was opened so the path could be repaired, which put the facility into a temporary single-feed posture on feed A. That is normal practice. It is also the posture that removed the margin.
The PLC was watching the wrong side of the switch
When the utility provider lost feed A to an underground cable fault at 06.53 BST, the PLC was supposed to notice and start the generator plant. It did not. Google’s investigation found the undervoltage sensing inputs were wired upstream, on the line side of the isolated feeder breakers, and the logic checked for voltage there before commanding a start.
Feed B was still energized upstream of its open breaker. The active and standby master PLCs, running identical control logic, read that live voltage through shared sensors on the disconnected path. Their conclusion was that a healthy utility source was available. Their conclusion was wrong, and the batteries could not argue with it.
The onsite team tried a remote load transfer and the system blocked it, because the same false reading said feed B was energized. By the time engineers manually moved the load to the standby utility source, the batteries were spent. Hosts, storage and network hardware in the zone lost power, and services across the region absorbed the redirected traffic. Recovery ran until 13.08 BST.
A second PLC with the same logic is not a second opinion
This is the part worth stealing for your own site. Dual controllers are sold as resilience, and they are, against a controller dying. They are not resilience against a shared input being wrong. Two brains reading one lying sensor through the same upstream taps agree with each other, and the disagreement never surfaces until the load drops.
The maintenance posture did the rest. Single-feed operation is legitimate, but it moves the risk from losing a feed to losing automatic transfer, and those failure modes get tested by different things. A generator load test verifies the plant starts. It does not verify the plant is asked to start by logic that can still see the truth. The 14 October test found a reason to open feed B and never tested what the PLC would conclude with it open.
What to check in your own facility this week
Ask where your generator start signal samples voltage, and whether that point is upstream or downstream of a breaker that gets opened for maintenance. Ask whether your two controllers use separate sensing, or share taps and identical logic. Ask what your last maintenance posture did to automatic transfer, and whether anyone tested the transfer logic in that posture rather than the generator alone. A facilities engineer can answer all three in an hour, and the answers are cheaper than the one Google got.
Google has its own list. Feed A cable repair is estimated for 13 November, and the site returns to a normal two-feed configuration by 17 November. The item that should interest everyone else is the plan to validate PLC configurations at other sites with similar topologies and PLC manufacturers, estimated for 15 November. A hyperscaler checking sibling sites for the same wiring choice is telling you the pattern is not unique to one building.
Availability zones are supposed to contain this kind of failure, and the boundary did hold at the metal. The blast radius still reached the region, because workloads failing over into neighboring zones saturated the capacity waiting for them. Physical redundancy is purchasable. Verified logic is not, and it does not ship in the box.
Related reading. Google Bought the Power Plant Before It Built the Data Center and A Converted Aircraft Engine Beats the Grid by Six Years.
Get the next one before it is old news
Independent analysis of cloud-native infrastructure, Kubernetes and data center economics. No vendor spin.