Approaches
Incident Management
A defined way of detecting, handling and learning from production failures, agreed before an outage so that nobody has to invent a process in the middle of one.
Part of Maintenance & Modernization
Incident management is the agreed process for detecting a failure or slowdown in a live system, responding to it, keeping people informed and learning from it afterwards.
The parts of a response
- Monitoring and alerting that tell the team about a problem before a customer reports it.
- One named coordinator, so several people do not investigate the same thing while nobody updates the business.
- Regular updates to stakeholders on what is affected and when the next update will come. People often need this more than the technical detail.
- Restoring service first. Finding the root cause can wait, as long as logs and other evidence are kept for the diagnosis afterwards.
What speeds up recovery
How quickly a team recovers depends on work done beforehand: runbooks for common failures, access to logs and dashboards, and a tested way to roll back a release.
The review afterwards
After a significant incident we hold a review that looks for causes without assigning blame, because blame discourages people from reporting problems early. The review asks what allowed the failure and what would detect it sooner, and its actions go into the backlog. A bug found in production becomes an automated test, so a repeat of it is caught before release.