Designing for availability-zone failure

Spreading a service across three availability zones is the first step most teams take toward resilience, and often the last. The architecture diagram shows three boxes, the load balancer spans all of them, and everyone assumes a zone failure would be absorbed.

Zone failures are rarely clean. A zone can lose power, lose network connectivity to its neighbors, or degrade slowly so that health checks pass while requests time out. Whether a multi-zone service survives depends on decisions that the diagram does not show: how much capacity each zone holds, what the system needs to do during recovery, where the data lives, and whether anyone has tried it.

Static stability

The most useful principle comes from the AWS Builders' Library: a system is statically stable if it keeps working through a failure without having to make any changes. No new instances, no DNS updates, no configuration pushes.

This matters because the things you would do to react to a zone loss are exactly the things that are under stress during one. Launching replacement capacity depends on the cloud provider's control plane, which every other customer in the region is also calling. Instance types can be temporarily unavailable in the remaining zones. Autoscaling reacts in minutes, and the incident is happening now.

A statically stable design already has what it needs:

• Enough running capacity in the remaining zones to carry the full load.
• Health checks that remove the failed zone from traffic automatically.
• Data that is already replicated, with failover that does not need a human.
• No dependency on the failed zone for anything in the request path.

Capacity per zone

If a service needs a certain capacity at peak and runs across N zones, losing one zone leaves N minus 1 to carry everything. Each zone must be sized for its share of that:

Three zones is the usual balance. Two zones means running double capacity all the time, and losing one leaves no redundancy at all.

On Kubernetes, capacity per zone comes from the node groups and from where pods land. Spread replicas evenly so that each zone holds its share:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: checkout

Then size replicas so that the survivors can carry peak. A Deployment autoscaled to six replicas at peak, two per zone, needs to run nine to be statically stable, or have an autoscaler and node capacity you have proven can react in time. Cluster Autoscaler should run one node group per zone, with --balance-similar-node-groups, so scale-ups do not concentrate in one zone.

Shared infrastructure that is secretly zonal

Many single points of failure sit below the application:

• NAT gateways are zonal resources. One NAT gateway for the whole VPC means a zone failure takes outbound internet access from every zone. Run one per zone, with each private subnet routing to its own.
• Single-instance tools such as a self-hosted CI runner, a bastion host or a license server.
• Interface VPC endpoints with a network interface in only one subnet.
• Stateful sets with zonal volumes. An EBS volume belongs to one zone, so a pod that needs it cannot reschedule elsewhere.

Walking through the infrastructure code for anything created once, without a zone loop or a list of subnets, finds most of these.

Load balancing during a zone failure

Load balancers health-check targets and stop sending traffic to unhealthy ones. The details determine how quickly and how cleanly that happens.

On AWS, Application Load Balancers have cross-zone load balancing on by default: each load balancer node sends traffic to targets in all zones. Network Load Balancers have it off by default: each node sends only to targets in its own zone, and DNS stops returning the node's address if all its targets fail.

The harder case is a zone that is degraded rather than down. Targets pass shallow health checks but fail real requests, often because a dependency in that zone is slow. Two mitigations:

• Deeper health checks, carefully. A health check that tests downstream dependencies can take every target out at once if a shared dependency fails. Check what is local to the instance, and alert on the rest.
• Zonal shift. Amazon Route 53 Application Recovery Controller can shift traffic away from one zone for a supported load balancer with a single API call, and zonal autoshift lets AWS do it when it detects a zone impairment. Review the prerequisites for your load balancer configuration before relying on it.

aws arc-zonal-shift start-zonal-shift \
  --resource-identifier arn:aws:elasticloadbalancing:eu-west-1:123456789012:loadbalancer/net/checkout/8f2c1d0e9a7b6c5d \
  --away-from euw1-az2 \
  --expires-in 2h \
  --comment "checkout latency in euw1-az2"

The command uses zone IDs such as euw1-az2, not names such as eu-west-1b, because zone names map to different physical zones in different accounts.

Data replication

Stateless tiers are the easy part. The data tier decides how long recovery takes and whether anything is lost.

• Managed relational databases: RDS Multi-AZ keeps a synchronous standby in another zone and fails over automatically, typically in one to two minutes. Applications must reconnect, so check that connection pools handle a dropped connection and a DNS change. Aurora replicates storage across three zones and promotes a replica on failure.
• Caches: ElastiCache for Redis with Multi-AZ and automatic failover promotes a replica. Without it, a cache in the failed zone is simply gone, and the database behind it takes the full load at the worst moment.
• Self-managed quorum systems: etcd, ZooKeeper, Consul and similar need a majority. Three members in three zones survive one zone loss. Three members in two zones do not, if the zone holding two of them fails.
• Kafka: set broker.rack to the zone, a replication factor of 3 and min.insync.replicas=2, so every partition has a copy outside any single zone and writes still succeed with one zone gone.

Synchronous replication across zones costs latency and cross-zone data transfer. That cost is the price of not losing writes, and it belongs in the design decision, not as a surprise on the bill.

Testing a zone loss

A design that has not been tested is a hypothesis. Start small and increase realism:

1. Drain a zone in staging. Cordon and drain every Kubernetes node in one zone, or scale that zone's node group to zero, and watch latency, errors and failovers.
2. Shift traffic away from a zone in production with a zonal shift during working hours. This tests capacity and load balancing without touching data.
3. Simulate the failure itself. AWS Fault Injection Service has an AZ power interruption scenario that stops instances, fails over databases and blocks network traffic in one zone. Run it in staging first.

Each test should answer concrete questions. Did the remaining zones carry peak without autoscaling? How long did the database failover take, and how many requests failed during it? Which alerts fired, and did any of them help?

Where to start

1. Calculate per-zone capacity for each critical service and compare it with what actually runs at peak.
2. Find zonal single points of failure, starting with NAT gateways and anything created once in infrastructure code.
3. Add zone topology spread to every multi-replica workload.
4. Confirm database and cache failover is enabled and that clients reconnect after it.
5. Run a zone drain in staging, then a zonal shift in production, and write down what broke.
6. Repeat the test after major architecture changes, not just once.