Migrating from Text Logs to Structured JSON

Grep across pod logs worked at 3 services. At 30, it did not.

Before

[2024-07-01 14:32:11] INFO Request completed in 45ms

Unqueryable in Loki. No trace_id. No user_id. Correlation meant SSH and pray.

After

{
  "@t": "2024-07-01T14:32:11Z",
  "@l": "Information",
  "@mt": "Request completed",
  "elapsed_ms": 45,
  "trace_id": "abc123",
  "route": "/api/v1/orders"
}

Rollout Plan

1. Add structured logging behind feature flag — dual-write text + JSON for 2 weeks
2. Update Grafana dashboards to JSON field queries
3. Remove text formatter per service after dashboard parity
4. Enforce via lint rule: no string interpolation in log messages

MTTR on production incidents dropped ~35% in the first quarter after migration.