How to Plan for a Cloud Region Outage: Lessons From the AWS us-east-1 and Google Cloud Failures

A single cloud region going dark can take banking apps, smart doorbells, airline check-in and internal tools offline at the same time. Two of the largest failures of recent years show why. On October 19 and 20, 2025, Amazon Web Services lost DynamoDB name resolution in its Northern Virginia region (us-east-1) for close to three hours, and knock-on problems with new server launches and load balancers stretched the event to about 14.5 hours. In June 2025, a bad policy update crashed a core API-checking service across Google Cloud, with the worst-hit region taking up to roughly 2 hours 40 minutes to fully recover.

Neither failure was caused by a fire or a flood. Both came from software and automation. That matters for anyone deciding how much resilience to buy, because it means "the data center is fine" is not the same as "my app is up."

This guide explains what failed in those two incidents, what the providers' own reliability guidance recommends, and how to choose and test a recovery strategy that fits your budget.

Key facts at a glance

Who this guide is for

This is for engineering leads, platform teams, startup founders and IT managers who run customer-facing services on AWS, Google Cloud or Azure and want a plain-English way to decide how far to go beyond a single region. It assumes you know what a region and an availability zone are, but not that you have a formal disaster recovery plan.

What actually failed in the AWS outage

AWS said the root cause was a latent race condition in the system that manages DNS records for DynamoDB. That system has two parts: a planner that builds new DNS plans and an enactor that applies them through Route 53, running independently in three availability zones.

One enactor was badly delayed while applying an old plan. Meanwhile a second enactor applied a much newer plan and then ran a clean-up job that deletes old plans. The delayed enactor overwrote the newer plan with its stale one just as the clean-up deleted that stale plan, which removed every IP address for the regional endpoint. The system was left in a state that blocked further automatic updates and needed manual repair.

The effects spread because so much depends on DynamoDB. AWS said new EC2 instance launches failed with "request limit exceeded" or "insufficient capacity" errors, Network Load Balancers saw connection errors between 5:30 AM and 2:09 PM PDT, and services such as Lambda, STS, Redshift, Amazon Connect and console sign-in for IAM users were all affected. Existing EC2 instances that were already running stayed healthy throughout.

Two details stand out for planners. First, customers using DynamoDB global tables could still read and write their replica tables in other regions, though replication to and from us-east-1 lagged. Second, some sign-in paths for other regions also broke, because they depended on endpoints in us-east-1. AWS said it disabled the DNS planner and enactor automation worldwide pending a fix, and is adding controls to limit how much capacity a load balancer can remove when health checks fail.

What failed in the Google Cloud outage

Google said a new quota-checking feature was added to its Service Control system on May 29, 2025, without proper error handling and without a feature flag. On June 12, a policy change containing unintended blank fields was written to the regional Spanner tables that Service Control reads, and that data replicated globally within seconds. Service Control hit a null pointer and went into a crash loop in every region.

Google's engineers identified the root cause within about 10 minutes and had a kill switch fully rolled out within about 40 minutes. Recovery in large regions was slower because restarting tasks created a "herd effect" that overloaded the database they depend on, and the service lacked randomized exponential backoff. Google also said its own status page was down for roughly the first hour, and some customers' monitoring, which ran on Google Cloud, failed too, so they had no signal at all.

Google has committed to making Service Control fail open, propagating globally replicated data incrementally, putting all changes to critical binaries behind feature flags, and keeping its communication tools running even when its main cloud is down.

Lessons that apply to your own systems

The common thread: a fast, global, automated change reached everything at once, and recovery tools depended on the thing that broke.

1. Know your hidden regional dependencies. Map which services, identity paths and control planes your app touches in each region. The AWS event showed that a workload "in another region" can still rely on us-east-1 for sign-in or management.
2. Fail over using the data plane, not the control plane. AWS's disaster recovery guidance says data planes typically have higher availability design goals than control planes, so failover should not depend on launching new resources or changing configuration during an incident. That is exactly what broke when EC2 launches failed.
3. Use backoff with jitter in your own clients. Google's slow recovery came partly from synchronized retries. Your services should retry with randomized exponential backoff so they do not stampede a recovering dependency.
4. Keep monitoring and status pages outside the blast radius. If your alerting runs in the same region or provider as your app, you may lose visibility exactly when you need it.
5. Roll out config and data changes gradually. Both incidents involved changes that propagated everywhere quickly. Stage configuration, feature flags and data changes the same way you stage code.

Choosing a recovery strategy: the four standard options

AWS's Well-Architected reliability guidance describes four disaster recovery strategies, listed in increasing order of cost and complexity. The same ideas apply on any cloud.

The guidance stresses that you should not pick a strategy more stringent than the business needs, because each step up costs more to build and run. It also notes that replication alone does not protect against data corruption or deletion. You still need point-in-time backups.

How to apply this, step by step

1. Set RTO and RPO per workload. Ask the business owner how long the service can be down and how much recent data it can afford to lose. A checkout flow and an internal reporting tool rarely need the same answer.
2. Start with multi-AZ. Spreading a workload across availability zones within one region protects against events like power loss or a single facility failure, and it is the cheapest meaningful step.
3. Pick a multi-region strategy only where the numbers justify it. If your RTO is measured in hours, backup and restore with infrastructure as code may be enough. If it is minutes, look at pilot light or warm standby.
4. Automate the rebuild. Keep infrastructure in code templates so a recovery region can be stood up without hand-built steps. Copy machine images and backups across regions and, where possible, across accounts to guard against account compromise.
5. Turn on object versioning for critical storage. Versioning protects against accidental or malicious deletion, which replication can copy faithfully.
6. Test failover on a schedule. AWS's guidance says it is critical to regularly assess and test your plan. A failover that has never been exercised is a hope, not a plan.
7. Write the communication plan too. Decide in advance where you will post status updates if your primary provider is down, and who decides to fail over.

What to do during a provider outage

• Check the provider's status page and your own health dashboard, but assume both may lag.
• Avoid actions that depend on the impaired control plane, such as launching fresh capacity in the affected region.
• If you have a standby region, follow your runbook rather than improvising.
• Keep customers informed early, even if the only message is that you are investigating.
• After recovery, read the provider's post-incident summary and update your dependency map.

Takeaways

• The biggest recent cloud failures were caused by automation and configuration, not physical disasters.
• Hidden dependencies on one region, especially for identity and management, can defeat a multi-region design.
• Match your recovery strategy to agreed RTO and RPO targets instead of buying the most expensive option by default.
• Fail over with already-running resources, use backoff with jitter, and keep monitoring outside the blast radius.
• Test the plan regularly, because the first real outage is a bad time to discover it does not work.