Health Checks and Failover That Don't Make Outages Worse
    Architecture
    9 min read

    Health Checks and Failover That Don't Make Outages Worse

    Shallow vs deep checks, flapping thresholds, detection-time math, fail-open rules, and why failing over to a cold target can turn one outage into two.

    Will Schulz
    Will Schulz
    CEO

    Health checks are supposed to make systems more available. In practice, they are one of the most common ways a small problem becomes a large one. An eager check pulls healthy servers out during a traffic spike. A check that tests every dependency marks a whole fleet unhealthy because one database is slow. A failover rule sends all traffic to a standby that has never served production load, and it falls over too.

    None of this means you should skip health checks. It means you should design them with the same care as the request path, because they are part of the request path. This article covers the decisions that matter most: what a check should test, how many failures should trigger action, how long detection actually takes, and what should happen when everything looks broken at once.

    Liveness and Readiness Are Different Questions

    The AWS Builders' Library article on implementing health checks describes a spectrum, from basic liveness checks that confirm a process is running and accepting connections, to deeper checks that exercise dependencies [1]. The deeper you go, the more failures you catch, and the more false positives you create. The article states it directly: one of the challenges of building a good health check is to guard carefully against false positives [1].

    Kubernetes makes the distinction explicit with separate probe types. A failed readiness probe causes the EndpointSlice controller to remove the Pod's IP address from the matching Services, so the Pod stops receiving traffic but keeps running [2]. A failed liveness probe causes a restart. The Kubernetes documentation warns that liveness probes should truly indicate unrecoverable application failure, such as a deadlock, and that incorrect liveness probes can lead to cascading failures: containers restart under high load, client requests fail, and the remaining Pods absorb more work [2]. Its recommendation for apps with strict back-end dependencies is that the liveness probe passes when the app itself is healthy, while the readiness probe additionally checks required back-end services [2].

    Deep Checks That Cascade

    The failure mode to fear is a dependency check that fails everywhere at once. If every web server's health endpoint queries the same database, a database slowdown makes every web server look unhealthy simultaneously. The Builders' Library notes that dependency health checks can turn a soft dependency into a hard one, and recommends that load balancers remove servers only if they have some problem that is definitively local to that server [1]. Amazon teams often keep fast-acting load balancer checks local and rely on separate, centralized systems with thresholds and human escalation for dependency-level decisions [1].

    The Google SRE book chapter on cascading failures describes the same dynamic from the load side. Load balancing policies that avoid servers that have served errors can exacerbate problems further, because removing capacity increases the load on the remaining servers and starts a snowball effect [3]. Servers can also appear unhealthy to the load balancer without crashing, which reduces capacity just as effectively as a crash [3]. The Builders' Library adds that taking servers out of service during an overload can cause a downward spiral, which is why it recommends that servers prioritize answering health checks even when busy [1].

    Keep the load balancer check cheap and local: process up, static response served, local config sane. Monitor dependencies separately. If a dependency must be in readiness, cap how much of the fleet automation may remove, and page a human at the cap.

    Flapping and Hysteresis

    One failed request is not evidence of an unhealthy server. Networks drop packets and garbage collectors pause, so health check systems use consecutive-result thresholds.

    Route 53 defines its failure threshold as the number of consecutive health checks an endpoint must pass or fail to change status from unhealthy to healthy or vice versa [4]. Kubernetes probes default to a failureThreshold of 3 and a successThreshold of 1 [2]. Cloudflare monitors expose consecutive_up and consecutive_down fields to smooth status changes [5], and its API example uses consecutive_up of 3 and consecutive_down of 2 [6].

    Requiring more consecutive successes to return a target than failures to remove it gives you hysteresis: a half-working target does not oscillate in and out of rotation. Recovery is slower, which is usually the right trade.

    Detection Time Is Arithmetic

    You can estimate how long failover takes. Time to detect is roughly the check interval multiplied by the failure threshold, plus the per-check timeout. For DNS-based failover, add the record TTL, because RFC 1035 defines TTL as the time interval in seconds that a resource record may be cached [7]. Resolvers holding the old answer keep sending users to the failed endpoint until it expires.

    Route 53 lets you choose a request interval of 10 or 30 seconds [8], with the 10-second option carrying an additional charge [4]. With a 30-second interval and a threshold of 3, detection takes around 90 seconds. Route 53's DNS best practices note that a TTL of 60 or 120 seconds is a common choice for records that are part of a rapid failover mechanism [9], and that propagation of record changes typically takes less than one minute globally [9]. Add a 60-second TTL and many users will see two to three minutes between failure and recovery.

    Inside a cluster, Kubernetes defaults of periodSeconds 10 and failureThreshold 3 [2] pull an unready Pod in about 30 seconds plus timeouts.

    A 60-second check interval sits at the slower end. With three consecutive failures required, detection alone takes about three minutes, and DNS caching adds more. That is appropriate for many workloads: internal tools, marketing sites, batch APIs, and services with a client-side retry layer that absorbs short blips. It is also the floor for some plans; Cloudflare lists a minimum monitor interval of 60 seconds on Pro, 15 on Business, and 10 on Enterprise [6]. It is not appropriate when your availability target cannot tolerate several minutes of errors per incident, or when failover is your only mitigation. In those cases, use faster in-region checks at the load balancer and treat DNS failover as the slower, coarse-grained backstop.

    Faster is not free. Route 53 notes that with a 30-second interval an endpoint receives a check about every two seconds on average across its health checkers, and with 10 seconds, more than once per second [4]. Cloudflare warns that low interval values can cause problems [5].

    Judge Health From Many Places

    A check run from a single location tells you whether that location can reach your service, not whether your users can. Route 53 aggregates results from health checkers around the world and considers an endpoint healthy if more than 18% of checkers report it healthy, a value chosen so that network conditions isolating some checker locations do not mark the endpoint down [8]. You must specify at least three checker regions if you customize them [4]. Cloudflare sends three probes from each configured region, considers a region healthy if the majority of its data centers pass, and considers the endpoint healthy if the majority of regions are healthy [5].

    Apply the same idea to your own synthetic monitoring. The SRE book's chapter on monitoring distributed systems describes black-box monitoring as testing externally visible behavior as a user would see it, useful for paging on problems that are ongoing and affecting users [10]. Pair synthetic checks from multiple vantage points with the four golden signals, latency, traffic, errors, and saturation [10], measured on real traffic. A probe that passes while error rates climb is testing the wrong thing.

    When Everything Looks Unhealthy

    Decide in advance what happens when every target fails its check at once. Usually that means the check is wrong, not that every server died simultaneously. Several systems fail open in this case. The Builders' Library notes that the AWS Network Load Balancer fails open if no servers are reporting as healthy [1]. Route 53's rule is that if no record in a group is healthy, all records are treated as healthy and selected by routing policy; for failover records where both primary and secondary are unhealthy, Route 53 returns the primary [11]. Cloudflare routes to a designated fallback pool whose health is not taken into account when all pools are unhealthy [12].

    Know which behavior your stack has, and make sure fail-open lands somewhere sensible. Serving a degraded response is usually better than serving nothing; the SRE book's chapter on handling overload recommends serving degraded responses that contain less data than normal rather than failing outright [13].

    Do Not Fail Over to a Cold Target

    Failover moves load, and the destination has to be able to carry it. The SRE book's example is two clusters normally handling 1,000 QPS each; when one fails, the survivor receives 1,200 QPS, cannot handle it, and starts running out of resources [3]. It also notes that processes are often slower immediately after starting than in steady state, and that a service not provisioned for a cold cache is at greater risk of outages [3]. Its advice is to increase load on a cluster slowly [3].

    For active-passive designs, that means keeping the standby warm with a slice of real traffic, provisioning it for full load (not "we'll autoscale"), and preferring gradual weight shifts over a single cutover when time allows. Kubernetes startup probes help on the instance level: they hold off liveness and readiness checks until initialization completes, so slow-starting containers are not killed mid-boot [2].

    The SRE book also recommends load testing components until they break and notes that gradually ramping load can produce different results than jumping straight to expected levels [3]. Run scheduled failover drills, compare real detection and recovery time with your arithmetic, and fix the surprises.

    How AW3 Host Fits

    AW3 Host provides failover across the providers you deploy to, using 60-second health checks, alongside weighted and geo routing and canary or blue-green rollouts with rollback. By the arithmetic above, that suits cross-provider failover where a few minutes of detection is acceptable, and weighted routing lets you shift traffic gradually instead of cutting over at once. It does not replace fast in-region load balancer checks or Kubernetes probes, and it cannot warm a cold standby or design your health endpoint; those remain your decisions.

    Takeaways

    Keep liveness and load balancer checks shallow and local, and cap how much capacity automation can remove. Use consecutive thresholds, with recovery slower than removal. Compute detection time as interval times threshold plus timeout plus TTL. Check from several regions and pair probes with golden-signal metrics. Know your fail-open behavior. Keep failover targets warm, ramp traffic gradually, and drill the path so the first real failover is not the first test.

    Sources

    1. Implementing health checks — Amazon Builders' Library (AWS), 2019
    2. Liveness, Readiness, and Startup Probes — Kubernetes Documentation, 2026
    3. Addressing Cascading Failures — Google, Site Reliability Engineering, 2016
    4. Values that you specify when you create or update health checks — Amazon Web Services, 2026
    5. Load Balancing: Monitors — Cloudflare Docs, 2026
    6. Create a monitor — Cloudflare Docs, 2026
    7. RFC 1035: Domain Names - Implementation and Specification — IETF, 1987
    8. How Amazon Route 53 determines whether a health check is healthy — Amazon Web Services, 2026
    9. Best practices for Amazon Route 53 DNS — Amazon Web Services, 2026
    10. Monitoring Distributed Systems — Google, Site Reliability Engineering, 2016
    11. How Amazon Route 53 chooses records when health checking is configured — Amazon Web Services, 2026
    12. Load Balancing: Health details — Cloudflare Docs, 2026
    13. Handling Overload — Google, Site Reliability Engineering, 2016
    Health ChecksFailoverReliabilityDNS
    Will Schulz
    Will Schulz
    CEO

    Writing about cloud infrastructure, DevOps best practices, and the future of application deployment. Follow for weekly insights on multi-cloud strategies and platform engineering.

    Stay Updated

    Get the latest articles on cloud infrastructure, DevOps, and platform engineering delivered to your inbox weekly.