Debugging High Load with strace and perf

The dashboard showed 2.1s P99. CPU at 22%. Memory flat. Classic “where is the time going?” problem.

Symptoms

• Latency correlated with write volume, not read volume
• Only affected one AZ
• No DB slow query log entries

Investigation

# Sample syscalls for 30 seconds on a hot pod
strace -c -p $(pgrep -f Gateway) -f

# Result: 68% time in fsync, 12% in write

The app was logging synchronously to a mounted volume. That AZ had slower EBS throughput after a maintenance window.

Fix

• Switch file logging to async stdout → Fluent Bit → centralized store
• Remove volume mount for app logs entirely
• Add alert on fsync latency via node exporter