Production Observability Setup
SLO & Alerting Design
Fewer alerts, each of which means something.
- Turnaround
- 1–2 week sprint
- Who it is for
- Teams with an on-call rota people dread.
01 — The problem
Alert fatigue is a design failure, not a discipline failure. When everything pages, nothing does — and the one alert that mattered was muted months ago.
02 — What is included
Exactly what you get, and nothing vague.
- SLIs defined from what users actually experience, not what is easy to measure
- SLO targets set with the error budget consequences agreed up front
- Burn-rate alerting to replace static threshold alerts
- Alert inventory: what to delete, what to downgrade to a ticket
- Routing to Slack or PagerDuty with escalation that matches severity
03 — What this usually finds
The recurring ones, named up front.
- Alerts on CPU rather than on user-visible symptoms
- Pages for conditions that self-resolve before anyone logs in
- No error budget, so reliability arguments stay subjective
- Severity levels that everybody ignores
Listed because they recur, not because they are guaranteed. The audit reports what is actually there.
Next step
Twenty minutes to find out whether this is worth doing.
Every engagement runs under a signed mutual NDA with a read-only IAM role. If the audit would not pay for itself, I will say so on the call.