Engineering incident reviews must lead to system changes, Heap argues
The engineering executive says clear ownership and better alerts can make corrective actions stick, while some risks may be consciously accepted.
Engineering postmortems can explain how an incident unfolded without reducing the chance it will happen again. In an essay published on Hacker News, engineering executive Michael Heap argues that reviews should focus on what will change, rather than treating a reasonable account of past decisions as sufficient progress.
Heap recounts being interrupted while explaining an incident to an engineering counterpart and the company’s SVP of Engineering. The executive, in Heap’s telling, said they did not want the details and wanted to know what would change. Heap says he came to understand the response as an expression of trust in the people involved, coupled with an expectation that the system would be improved.
He argues that explanations can defuse pressure to act: once a sequence of decisions seems understandable, teams may move on without addressing the conditions that produced it. Reconstructing timelines and decisions has value, but understanding an incident is not the same as fixing it.
Heap points to specific changes such as making ownership clear when someone is unavailable, planning for late changes to requirements, and improving alert quality so on-call engineers can identify important signals. Vague commitments to communicate better or be more careful, he says, are not durable corrective actions.
A useful test, in Heap’s view, is whether a fix would still work if the people involved left the organisation. Processes that force a decision at a critical point can help; systems that prevent a class of mistake are stronger still.
Heap also cautions against adding a process after every failure. If preventing a recurrence costs more than the likely harm, an organisation may choose to accept the risk, but should do so consciously rather than relying on promises to try harder.
