Zero-Downtime Deployments on Kubernetes

Rolling updates work until they don’t. WebSocket connections, gRPC streams, and in-flight HTTP requests all break when pods terminate too aggressively.

The Setup

Our API gateway handles ~12K req/s with average connection duration of 45 seconds. Standard RollingUpdate with maxUnavailable: 25% caused 502 errors during every deploy.

graph LR
  A[Load Balancer] --> B[Blue v1.2]
  A --> C[Green v1.3]
  B --> D[Drain Connections]
  C --> E[Accept Traffic]
  D --> F[Terminate Pod]

Solution: PreStop Hook + Readiness Probe

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15 && /app/drain"]
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 5
terminationGracePeriodSeconds: 60

The drain script:

1. Marks pod as not-ready (removed from service endpoints)
2. Waits for active connections to finish (max 45s)
3. Signals graceful shutdown

Lessons

• Always measure connection duration before choosing drain timeout
• terminationGracePeriodSeconds must exceed drain time + buffer
• Monitor kube_pod_status_ready during deploys