Idempotency Keys in Distributed APIs

When clients retry failed requests, without idempotency you get duplicate side effects — double charges, duplicate orders, orphaned records. This is one of those problems that seems simple until you’re debugging it at 2 AM.

The Problem

HTTP retries are inevitable. Mobile networks drop. Load balancers timeout. Clients implement exponential backoff. Your API must handle the same request arriving twice without corrupting state.

Design

We chose a client-generated idempotency key stored in Redis with a 24-hour TTL:

public async Task<IResult> HandlePayment(
    PaymentRequest request,
    IdempotencyService idempotency,
    CancellationToken ct)
{
    var key = request.Headers["Idempotency-Key"];
    if (string.IsNullOrEmpty(key))
        return Results.BadRequest("Idempotency-Key header required");

    var cached = await idempotency.GetAsync<PaymentResult>(key, ct);
    if (cached is not null)
        return Results.Json(cached, statusCode: cached.StatusCode);

    var result = await ProcessPayment(request, ct);
    await idempotency.StoreAsync(key, result, TimeSpan.FromHours(24), ct);
    return Results.Json(result);
}

Key Decisions

1. Redis over PostgreSQL — sub-millisecond lookups at gateway scale
2. 24-hour TTL — covers retry windows without unbounded storage
3. Store full response — including status code and body, not just success/failure
4. Key scope per endpoint — POST /payments keys don’t collide with POST /transfers

Edge Cases We Hit

• Concurrent duplicate requests: Two identical requests arriving simultaneously. Solved with Redis SET NX as a lock during processing.
• Partial failures: Payment succeeded but response lost. Store result atomically before returning.
• Key reuse with different payloads: Return 422 Unprocessable Entity if same key, different body hash.

Results

After deployment:

• Zero duplicate transactions in 6 months
• ~0.3ms added latency (Redis lookup)
• Idempotency key header adoption: 99.7% of POST requests within 2 weeks