In an outage the technical problem is rarely the hard part. Coordination is. This is the sequence that keeps a small team from making it worse.
Most small teams handle their first serious outage badly — not through incompetence, but because everyone debugs simultaneously, nobody talks to customers, and two people deploy conflicting fixes. The playbook below exists to prevent that.
The first ten minutes
- Declare it. Say the word "incident" out loud in the team channel. Ambiguity about whether this is serious wastes more time than any technical step.
- Name an incident lead. One person, explicitly. They do not debug — they coordinate, decide, and keep the timeline.
- Assess blast radius: who is affected, is money moving incorrectly, is data at risk. This determines everything that follows.
- Stop the bleeding before finding the cause. Roll back, disable the feature, put up maintenance. Understanding can wait; customer impact cannot.
- Post a status message. Even "we are investigating" beats silence — silence is what turns an outage into a trust problem.
Roles, even in a team of four
| Role | Does | Does NOT |
|---|---|---|
| Incident lead | Decides, coordinates, tracks the timeline | Debug — the moment they do, coordination stops |
| Investigator | Finds and fixes the cause | Talk to customers |
| Comms | Updates status page, customers, internal team | Speculate about cause publicly |
In a small team one person may hold two roles — but incident lead and investigator must never be the same person during a serious incident.
What to say to customers
- Acknowledge quickly, even without answers — "we are aware and investigating" within minutes
- State impact in their terms: what they cannot do right now, not which service is degraded
- Give a next-update time, and honour it even if nothing has changed
- Never speculate on cause while it is unconfirmed — retractions cost more trust than silence would have
- Say clearly when it is resolved, and follow with a written explanation if the impact was material
After: the part everyone skips
A postmortem within 48 hours, while memory is accurate. Blameless — not as a nicety, but because blame makes people hide information and the next incident becomes harder to prevent.
- Timeline: what happened, when, who did what — facts only
- Impact: duration, affected users, money or data consequences
- Contributing factors, plural — single root cause is nearly always a simplification
- Action items with owners and dates; items without both are decoration
- What went well — the detection or response that worked is worth reinforcing
You do not rise to the level of your incident response plan. You fall to the level of the one you have actually practised.
Preparing before it happens
- Alerting on symptoms customers feel (checkout failing), not just infrastructure metrics (CPU high)
- A rollback that is one command and has been rehearsed in calm conditions
- A status page that exists before you need it
- Written escalation: who gets called at 3am, and who if they do not answer
- One practice run — deliberately break something in staging and follow the playbook
Use a recovery gate, not a green dashboard
A falling error rate can hide broken customer journeys or inconsistent data. Define what recovery must prove before closing the incident.
- Record impact, last known good state and recent changes.
- Choose a reversible action with an owner and a stop condition.
- Verify core journeys, delayed jobs and data reconciliation.
Frequently asked questions
What is the first thing to do when production goes down?
Declare an incident and name one person as incident lead, then mitigate before diagnosing — roll back, disable the broken feature, or enable maintenance mode. Restoring service comes first; understanding the cause is a task for after customers are working again.
Should we tell customers immediately?
Yes. Acknowledge within minutes, describe the impact in terms of what they cannot do, and commit to a next update time. Silence during an outage damages trust more than the outage itself, and speculation you later retract is worse than both.
Do we need a postmortem for every incident?
For anything with customer impact, yes — and it should be written within 48 hours while recollection is accurate. Keep it blameless: teams that assign blame get less honest information, which makes the next incident more likely, not less.
Should we roll back every deployment linked to an outage?
No. Check database and external side effects first. A rollback that cannot undo data changes may worsen the incident; choose a recovery action with explicit prerequisites.
In an incident right now?
We do emergency technical response — triage, mitigation, root cause, and the postmortem afterwards.
Further reading
Postmortem Best Practices: Writing One People Actually Read
Most postmortems are archaeology: an accurate record of something nobody will change. A useful one produces a small number of things that actually get done.
MVP Technical Audit: What We Check in the First 48 Hours
Most MVP audits produce a document. A useful one produces decisions: what is on fire, what can wait, and what it costs to fix.