My monitoring alert fired at 1:47 AM. API response times jumped from 200ms to 12 seconds. Users in Europe were complaining on Twitter. Here's how I found and fixed it.
Step 1: Don't Panic-Deploy
My first instinct was to roll back. I resisted. The last deploy was 6 hours ago and traffic had been fine until 30 minutes ago. Something else changed.
Step 2: Check the Obvious
- Database connections: maxed out (found it)
- CPU/Memory: normal
- Third-party API status: all green
- Recent config changes: none
Root Cause
A cron job I added last week ran a full table scan every hour. At low traffic it was invisible. A viral tweet spiked concurrent requests and the scan locked the table.
Fix and Follow-Up
Disabled the cron, added an index, re-enabled with a query rewrite. Next day: added connection pool limits and slow-query alerts.
Production teaches lessons that localhost never will.