HPA with Custom Metrics: Scaling Beyond CPU

The Incident

Black Friday traffic spike. CPU stayed at 40% but latency went from 50ms to 2.3s. HPA didn’t scale because CPU wasn’t the bottleneck — thread pool exhaustion was.

Fix: Custom Metrics HPA

Installed prometheus-adapter and exposed request rate + queue depth:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gateway-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gateway
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "500"
    - type: Pods
      pods:
        metric:
          name: threadpool_queue_length
        target:
          type: AverageValue
          averageValue: "10"

Timeline

1. T+0: Latency spike detected via alerting
2. T+5m: Manual scale to 20 replicas (CPU still ~45%)
3. T+15m: Latency normalized
4. T+2d: Custom metrics HPA deployed
5. T+1w: Load test validated auto-scale at 500 req/s/pod threshold

Now HPA reacts within 30 seconds of traffic increase, before latency degrades.