On October 19 and 20, 2025, a single empty DNS record in AWS's US East (N. Virginia) Region turned into roughly fourteen and a half hours of disruption. AWS's own summary dates the event from 11:48 PM PDT on October 19 to 2:20 PM PDT on October 20, and the list of affected services reads like a dependency graph of the region: DynamoDB, EC2 launches, Network Load Balancer, Lambda, ECS, EKS, Fargate, STS, IAM sign-in, Redshift, Amazon Connect and the AWS Support Center [1].
Afterwards, many teams asked whether they should be multi-cloud. Some of what broke would have been routed around by a second provider. Much of it would not, and some popular failover designs failed precisely because they depended on the thing that was down. This post uses AWS's postmortem to separate lessons that need a second provider from ones that just need better architecture inside the first.
What Actually Broke
DynamoDB manages its regional DNS with a DNS Planner, which produces plans for which load balancers back an endpoint, and three redundant DNS Enactors that apply them to Route 53. According to the postmortem, a latent race condition between two of them let a delayed Enactor apply an older plan over a newer one, after which the other Enactor's clean-up process deleted that older plan. The result, in AWS's words, was an incorrect empty DNS record for the service's regional endpoint, dynamodb.us-east-1.amazonaws.com, and a system state that ultimately required manual operator intervention to correct [1].
An empty record is a nasty failure mode: the endpoint did not return retryable errors, it resolved to nothing. AWS identified the root cause by 12:38 AM, restored DNS by 2:25 AM, and DynamoDB recovered by 2:40 AM PDT [1]. That alone would have been a three-hour incident. It was not the end.
The Cascade: Control Planes Fail Slowly
EC2's DropletWorkflow Manager, which maintains leases on the physical servers that host instances, depends on DynamoDB. When its lease checks failed, the fleet's state drifted, and new instance launches failed with either a request limit exceeded or an insufficient capacity error [1]. Recovery required throttling, host restarts and a long backlog of network configuration propagation. EC2 did not fully recover until 1:50 PM PDT on October 20, about eleven hours after DynamoDB itself was fixed [1].
Then Network Load Balancer got involved. Between 5:30 AM and 2:09 PM PDT, NLB health checks failed against new instances whose network state had not yet propagated, which caused nodes to be alternately removed from and returned to DNS. AWS disabled automatic health check failover at 9:36 AM and re-enabled it at 2:09 PM [1]. That health check churn spilled back into other services: STS, which had recovered at 1:19 AM, saw elevated errors again between 8:31 AM and 9:59 AM because of NLB health check failures [1].
The pattern: data planes, which serve traffic, mostly recovered once DynamoDB's DNS was back. Control planes, which create, change and scale things, took much longer, because recovery itself required launching instances and propagating network state. AWS's static stability article in the Builders' Library draws exactly this line: a control plane is the machinery involved in making changes to a system, and a data plane is the daily business of those resources [2].
Hidden Regional Dependencies
Teams with nothing in us-east-1 still felt this, because some global-looking things are anchored there. The postmortem notes that customers signing in to the AWS Management Console with an IAM user saw increased authentication failures, and customers with IAM Identity Center configured in us-east-1 were unable to sign in through it [1]. If your break-glass procedure assumes an engineer can log into the console and click a button, that procedure had a dependency on the failing region.
DNS management is another one. AWS's Well-Architected reliability guidance states that the Route 53 management APIs and consoles run on control planes located in a single Region, US East (N. Virginia), and that those control planes are not included in the Route 53 SLA, while the data plane that answers queries and evaluates health checks is globally distributed and designed for a 100% availability SLA [3]. AWS's networking blog says the same of its global services generally [4].
The audit: list each action your failover runbook takes and which provider and region's control plane it calls. Console logins, role assumption via a regional STS endpoint, DNS edits, autoscaling changes, CI runners and regional secret stores all count.
Why Failover That Calls the Failing Control Plane Fails
The Well-Architected pillar lists three anti-patterns for recovery: depending on changing DNS records to re-route traffic, depending on control-plane scaling operations to replace impaired components, and relying on extensive multi-service control plane actions to remediate [3]. October 2025 is close to a worked example of all three. A runbook that said launch replacement capacity in us-east-1 hit EC2 launch errors for most of a day. A runbook that said edit the Route 53 record leaned on a control plane AWS itself tells you not to rely on for recovery [3].
The alternative is to make failover a data plane decision that was configured in advance. Route 53 failover records return the primary resource while its health check is healthy and the secondary when it is not; the health checks are run from eight AWS Regions, and by default an endpoint is considered healthy if at least 3 of 16 checkers report it as healthy [4]. Nothing in that path requires calling an API during the incident. The same logic applies to capacity: AWS's guidance is to provision sufficient capacity in the cluster used for failover rather than asking Auto Scaling to add instances mid-incident, and to create functions, load balancers and tables before a degradation, because creation is a control plane action [3].
Static stability is the name for that discipline. In a statically stable design, the overall system keeps working even when a dependency becomes impaired, and all capacity that will be needed in the event of an impairment is already fully provisioned [2]. It costs money up front so you need nothing from anyone during the outage.
There is also a failure mode on the other side. NLB's own health check failover amplified the incident by repeatedly pulling capacity in and out, which is why one of AWS's remediations is a velocity control that limits how much capacity a single NLB can remove when health check failures trigger AZ failover [1]. Your automation needs the same brake: cap traffic shifted per interval, require consecutive failures before acting, and page a human rather than flap.
DNS TTLs Set the Floor on Recovery Time
Whatever does your failover, clients only see the change when their cached answers expire. Route 53's documentation recommends a TTL of 60 seconds or less on records associated with a health check so that clients respond quickly to changes in health status, and notes that longer TTLs mean recursive resolvers keep using cached values for longer [5]. A one-hour TTL can mean an hour of traffic to a dead target.
Short TTLs raise query volume and cost, so use 60 seconds on records you intend to fail over and longer TTLs elsewhere. Also check long-lived client connections: a pooled connection to a dead load balancer will not re-resolve until something forces it.
What a Second Provider Would and Would Not Have Changed
A second provider helps most with stateless tiers. A frontend, API gateway or worker fleet already running elsewhere, behind routing that does not live in us-east-1, could have kept serving through the EC2 launch failures and NLB churn. The key word is already: creating a standby mid-incident is a control plane action on whichever provider you use.
A second provider helps much less with state. If your system of record was a DynamoDB table in us-east-1, the second provider's compute had nothing to read. The postmortem is instructive here: customers using DynamoDB global tables could connect to and issue requests against their replica tables in other Regions, though they saw prolonged replication lag to and from the us-east-1 replicas [1]. That is a multi-region data design inside one provider, and it did more for availability than any amount of cross-cloud compute without the data would have. Cross-provider state replication is possible but expensive, and forces you to decide in advance which writes you will lose or reconcile.
A second provider is also not a guarantee of independence. On November 18, 2025, Cloudflare had its own major outage: a database permissions change caused a Bot Management feature file to double in size, exceed a limit in its proxy software, and fail, with significant failures from 11:20 UTC and all systems normal by 17:06 UTC [6]. If your plan put one edge provider in front of everything as the neutral layer, that layer is its own single point of failure. Every provider you add is another dependency to map.
Finally, cost. A warm standby on a second provider means idle capacity, duplicated secrets and configuration, two deploy targets to keep in sync, and regular failover rehearsals. For many teams the better first step is a second region on their existing provider with pre-provisioned capacity and replicated data, adding cross-provider redundancy only where it is cheap: static assets, stateless APIs and status pages.
How AW3 Host Fits
AW3 Host is a deployment control plane. It can deploy the same stateless service to several of the 17 providers it supports, keep those deployments running as warm standbys, and route between them with weighted or geo routing and failover driven by 60-second health checks. Daily cost sync and budget policies make the standby's price visible. Canary and blue-green releases with rollback cover the other common outage trigger, your own bad deploy.
It does not replicate your database, so state locked in one region stays locked there. It is also a dependency itself, so static stability applies: standbys should exist before an incident, not be created during one. And like any health-check-driven failover, it is bounded by your DNS TTLs and threshold tuning.
Takeaways
The October 2025 event was an empty DNS record cascading through control planes, with recovery taking far longer than the original fault [1]. The durable lessons are architectural. 1. Map every control plane and region your recovery path depends on, including sign-in, STS and DNS management. 2. Make failover a data plane decision configured in advance, with pre-provisioned capacity [2][3]. 3. Rate-limit your own automation so it cannot amplify an incident [1]. 4. Set short TTLs on records you plan to fail over [5]. 5. Solve state first, usually with multi-region replication, before buying cross-provider compute. Multi-cloud failover is a real tool for stateless tiers, not a substitute for knowing where your data lives.
Sources
- Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region — Amazon Web Services, 2025
- Static stability using Availability Zones (Amazon Builders' Library) — Amazon Web Services, 2019
- REL11-BP04 Rely on the data plane and not the control plane during recovery (Reliability Pillar) — AWS Well-Architected Framework, 2026
- Creating Disaster Recovery Mechanisms Using Amazon Route 53 — AWS Networking & Content Delivery Blog, 2022
- Values specific for failover records (Amazon Route 53 Developer Guide) — Amazon Web Services, 2026
- Cloudflare outage on November 18, 2025 — Cloudflare, 2025

Writing about cloud infrastructure, DevOps best practices, and the future of application deployment. Follow for weekly insights on multi-cloud strategies and platform engineering.
