When DNS Failover Did the Opposite of Failover
TTL misconfiguration routed 40% of traffic to a decommissioned region for 18 minutes.
We flipped DNS to the secondary region during a primary outage. Traffic split 60/40 instead of 100/0.
Root Cause
- Old A record had TTL 86400 (24 hours) cached at major resolvers
- New record propagated to Route 53 immediately
- Resolvers that cached the old IP kept sending traffic to a drained load balancer
Timeline
| Time | Event |
|---|---|
| T+0 | Primary region declared unhealthy |
| T+2m | DNS updated to secondary |
| T+5m | 40% traffic still hitting primary LB (connection refused) |
| T+18m | Cache expiry on largest resolver ASNs |
| T+25m | Error rate normalized |
Run quarterly DNS failover drills. Updating Route 53 is not the same as the world seeing the update.
dns , incident , failover · AWS , Linux