Google’s postmortem template uses a Shakespearean sonnet example to bring it to life. Significant facts are outlined from the get-go, including date, authors, status, summary, impact, and root causes. The trigger, resolution, and detection are all defined on the first page, giving readers a big-picture understanding of the project in minutes.
The action items are broken down into a clear, concise table. The action type, owner, and bug are confirmed, ensuring everyone knows who is in charge of what.
The report concludes with a comprehensive section on lessons learned. Unpacking what went well, what went wrong, and where we got lucky is essential. The entire incident culminates in a step-by-step timeline, including timestamps and essential updates. Any supporting information is noted for future reference.
Postmortem Example
Date: 2015-10-21
Authors: jennifer, martym, agoogler
Status: Complete, action items in progress
Summary: Shakespeare Search down for 66 minutes during period of very high interest in Shakespeare due to discovery of a new sonnet.
Impact: Estimated 1.21B queries lost, no revenue impact.
Root Causes: Cascading failure due to combination of exceptionally high load and a resource leak when searches failed due to terms not being in the Shakespeare corpus. The newly discovered sonnet used a word that had never before appeared in one of Shakespeare’s works, which happened to be the term users searched for. Under normal circumstances, the rate of task failures due to resource leaks is low enough to be unnoticed.
Trigger: Latent bug triggered by sudden increase in traffic.
Resolution: Directed traffic to sacrificial cluster and added 10x capacity to mitigate cascading failure. Updated index deployed, resolving interaction with latent bug. Maintaining extra capacity until surge in public interest in new sonnet passes. Resource leak identified and fix deployed.
Detection: Borgmon detected high level of HTTP 500s and paged on-call.
Action Items:
Action Item | Type | Owner | Bug | |
Update playbook with instructions for responding to cascading failure | mitigate | jennifer | n/a DONE | |
Use flux capacitor to balance load between clusters | prevent | martym | Bug 5554823 TODO | |
Schedule cascading failure test during next DiRT | process | docbrown | n/a TODO | |
Investigate running index MR/fusion continuously | prevent | jennifer | Bug 5554824 TODO | |
Plug file descriptor leak in search ranking subsystem | prevent | agoogler | Bug 5554825 DONE | |
Add load shedding capabilities to Shakespeare search | prevent | agoogler | Bug 5554826 TODO | |
Build regression tests to ensure servers respond sanely to queries of death | prevent | clarac | Bug 5554827 TODO | |
Deploy updated search ranking subsystem to prod | prevent | jennifer | n/a DONE | |
Freeze production until 2015-11-20 due to error budget exhaustion, or seek exception due to grotesque, unbelievable, bizarre, and unprecedented circumstances | other | docbrown | n/a TODO |
2015-10-21 (all times UTC)
ManyHttp500s
from all clusters#shakespeare
, names jennifer incident commanderGoogle’s postmortem template uses a Shakespearean sonnet example to bring it to life. Significant facts are outlined from the get-go, including date, authors, status, summary, impact, and root causes. The trigger, resolution, and detection are all defined on the first page, giving readers a big-picture understanding of the project in minutes.
The action items are broken down into a clear, concise table. The action type, owner, and bug are confirmed, ensuring everyone knows who is in charge of what.
The report concludes with a comprehensive section on lessons learned. Unpacking what went well, what went wrong, and where we got lucky is essential. The entire incident culminates in a step-by-step timeline, including timestamps and essential updates. Any supporting information is noted for future reference.