Debugging a Cache Stampede at 3 AM

Timeline

02:47 — PagerDuty: Redis memory at 98%, connection count spiking
02:52 — PostgreSQL CPU at 100%, query queue depth 4,000
03:15 — Identified hot key product:catalog:v2 expiring
03:40 — Deployed probabilistic early expiration fix
04:10 — Systems stabilized

What Happened

A popular cache key expired. 2,000 concurrent requests missed cache simultaneously. Each triggered a 2-second PostgreSQL query. Redis filled with in-flight recomputation. Death spiral.

public async Task<T?> GetWithEarlyExpiration<T>(
    string key,
    Func<Task<T>> factory,
    TimeSpan ttl,
    double beta = 1.0)
{
    var cached = await _redis.GetAsync<CachedItem<T>>(key);
    if (cached is null)
        return await Refresh(key, factory, ttl);

    var delta = ttl.TotalSeconds * beta * Math.Log(Random.Shared.NextDouble());
    if (DateTime.UtcNow.ToUnixTimeSeconds() - cached.ComputeTime > ttl.TotalSeconds + delta)
        _ = Refresh(key, factory, ttl); // fire-and-forget refresh

    return cached.Value;
}

Prevention Checklist

• Probabilistic early expiration on hot keys
• Request coalescing (single flight pattern)
• Circuit breaker on cache miss → DB path
• Alert on cache miss rate, not just Redis memory