Designing for multi-region failover

Every multi-region proposal we've reviewed starts with a diagram: two regions, a load balancer on top, arrows pointing both ways labeled "sync." The diagram is correct and almost useless. It tells you nothing about what happens when the link between those two regions gets slow instead of down, which is the failure mode that actually shows up, over and over, in postmortems.

Replication lag is physics, not an engineering problem

Cross-region round trips run 60 to 150 milliseconds depending on the pair of regions, and that number doesn't move no matter how good your team is. Synchronous replication across that distance means every write waits on the slowest region, which is a latency budget most product teams won't accept once they feel it. So almost everyone ships asynchronous replication instead, and asynchronous replication means there is always a window of unreplicated data. The question isn't whether that window exists. It's how big it gets under load, and what you do with the rows sitting in it when you fail over.

We've watched teams discover their real replication lag for the first time during an actual regional outage, because their monitoring alerted on lag exceeding a threshold but nobody had load-tested what lag does during a traffic spike, which is exactly when failover tends to get triggered. Lag that's 200ms at steady state can be 40 seconds during a flash sale, and 40 seconds of unreplicated writes is not a rounding error for an order system or a ledger.

Split-brain is an organizational failure wearing a database's clothes

Split-brain gets described as a consensus problem, and mechanically it is: two regions both believe they're primary and both accept writes. But the actual cause in most incidents we've seen isn't a missing quorum algorithm. It's a health check that flapped during a network partition, an automated failover system that promoted a secondary while the primary was still reachable from some but not all clients, and no fencing mechanism to guarantee the old primary actually stopped accepting writes before the new one started. Fencing is the unglamorous part nobody wants to build: a way to forcibly cut off a demoted primary at the network or storage layer, not just tell it to stop and hope.

Automatic failover without fencing is worse than manual failover with a five-minute pager delay. We'd rather wake an engineer than let two regions independently accept conflicting writes for twenty minutes while an algorithm sorts out quorum.

What vendors leave off the architecture diagram

Cloud provider documentation for multi-region setups is honest about the happy path and quiet about the rest. A few things we've had to explain to clients after the fact, not before:

Standby capacity in the passive region has to be running at production scale to actually absorb a failover, not sitting at 10% so it's cheap. Teams that scale the secondary down to save money find out during the real event that it can't take the load, and the failover itself becomes the outage.

DNS-based failover is bounded by the TTL you set and by every resolver, ISP, and corporate proxy that ignores your TTL and caches longer anyway. Budget minutes, not seconds, for traffic to actually move, and don't put an SLA in front of a customer that assumes otherwise.

Connection pools, in-memory caches, and session state don't fail over with the database. A region that comes back healthy on paper can still serve a wave of cold-cache requests and exhausted connection pools that look like a second outage stacked on the first.

License costs for a warm standby often double your database spend, and procurement rarely flags this until the renewal, by which point the standby is load-bearing and can't be cut.

Choose active-passive on purpose, not by default

Active-active sounds better in a sales deck. For most enterprise systems we build, active-passive with deliberate, tested, partially manual promotion is the safer design, because it trades a few minutes of recovery time for the ability to reason about exactly one writer existing at any moment. Active-active earns its complexity only when the business genuinely cannot tolerate that recovery window and is willing to fund the fencing, conflict resolution, and testing that active-active correctness actually requires. Most businesses that ask for active-active haven't done that math. Ask them what their last failover drill looked like before you design around a capability they've never actually exercised.