Circuit Breakers for Downstream Dependencies
Polly integration, failure thresholds, and the half-open state that prevented a cascade outage.
When the inventory service slowed to a crawl, the gateway thread pool filled waiting on timeouts. Circuit breakers cut the feedback loop.
Configuration
services.AddHttpClient<IInventoryClient, InventoryClient>()
.AddPolicyHandler(Policy
.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
.Or<HttpRequestException>()
.CircuitBreakerAsync(
handledEventsAllowedBeforeBreaking: 5,
durationOfBreak: TimeSpan.FromSeconds(30)));States
- Closed: Normal calls through
- Open: Fail fast, return cached fallback or 503
- Half-open: Single probe request after break duration
Tuning
Failure threshold too low → flapping on transient blips. Too high → cascade before open. We started at 5 failures / 30s break, adjusted per dependency based on error budget.
During inventory outage: gateway stayed healthy. Orders queued for retry instead of timing out every client.
resilience , circuit-breaker , polly · .NET , Redis