During an incident, choose the action most likely to restore acceptable service with controlled risk.
During an incident, choose the action most likely to restore acceptable service with controlled risk. A rollback reintroduces a known application version, but it may not reverse data changes. A hotfix changes the current version and needs evidence that the proposed correction addresses the observed failure. Neither option is automatically safer.
Establish the change and compatibility boundary
Correlate the incident with deployments, configuration, migrations and provider changes. Check whether the previous version can read and write current data. A destructive schema change, irreversible external operation or new message format may make a simple application rollback unsafe. Preserve relevant evidence while prioritising containment of ongoing harm.
Compare the available recovery actions
Consider disabling a feature, changing traffic routing, reducing workload or restoring a compatible version before writing new code. For a hotfix, identify the narrowest verified cause and the tests needed to protect the affected journey. Compare time to execute and verify, blast radius, reversibility and the consequences of being wrong.
Execute one coordinated intervention
Assign an owner and announce the intended action and success criteria to responders. Avoid overlapping changes that make the result impossible to interpret. Use the established deployment path where possible and preserve an audit trail for emergency changes. If the action depends on a migration or data correction, treat that as a separate controlled step.
Verify recovery beyond a green deployment
Check real user journeys, error rates, queue progress and data integrity. A healthy process does not prove missed work has been recovered. Keep monitoring until the agreed stability window is reached, then schedule permanent repair and incident review. Record why the chosen action was appropriate with the information available, including options ruled out by compatibility constraints.
- Related service
- Application disaster recovery: test RTO and RPO
- A production runbook a small team can actually use
Frequently asked questions
Is rollback always faster?
No. Compatibility checks and data consequences can make a rollback slower or unsafe.
Can we roll back a database migration?
Only when its reverse path is valid for current data and has been tested. Application rollback alone may be insufficient.
When is a hotfix reasonable?
When the cause and correction are narrow, verifiable and lower risk than available alternatives.
Should multiple teams change things at once?
Coordinate interventions so their effects can be understood and conflicting changes avoided.
When can the incident close?
After agreed service and data checks pass, outstanding recovery work is owned and the stability condition is met.
Bring the scope. We will help make it buildable.
Share the user journey, integrations and launch constraints. We can clarify the scope and prepare an estimate with assumptions and exclusions.
Further reading
Application disaster recovery: test RTO and RPO
A disaster recovery plan is credible when the team can demonstrate restoration of a useful service.
A production runbook a small team can actually use
A runbook should help a responder move from a specific symptom to a safe decision.
Incident severity levels: a practical escalation matrix
Incident severity should describe current or credible business impact, not how alarming a log message looks.