Writing runbooks engineers actually follow during an incident

Pull up the runbook for your last serious incident and check the "last edited" date against the last time that service's architecture changed. In most engineering orgs we've worked with, that gap is measured in quarters, sometimes years. The runbook still tells you to restart a service that was decomposed into three services eight months ago, or points at a dashboard that was migrated to a different observability stack, or assumes a failover procedure that a since-departed engineer used to run by hand. Nobody deleted it. Nobody trusted it enough to follow it, either.

That's the real failure mode, and it's not a documentation problem. A wrong runbook is worse than no runbook, because it costs the responder time they don't have: read it, try step one, watch it fail, realize the whole document is suspect, then fall back to reading source code and querying logs cold, under a page from a VP asking for an ETA. We've sat in enough postmortem reviews to know the pattern by heart: "we had a runbook for this" shows up in the timeline right before "but it was wrong, so we improvised."

Staleness is a process failure, not an accident

Runbooks rot because nothing forces them to change when the system does. A pull request that changes a failover procedure gets reviewed and shipped; the runbook describing the old procedure sits untouched in a wiki nobody links from the PR template. Ownership is the other half of it. A runbook with no named owner is nobody's job to update, and "the on-call team" is not an owner, it's a rotation that turns over every two weeks. We tie every runbook to the service's owning team and require it to be touched as part of the change that invalidates it, the same way we'd require a migration script for a schema change. If a PR changes how a service fails over, the runbook update ships in the same PR or the PR doesn't merge.

Untested runbooks are hypotheses, not procedures

A runbook nobody has executed outside of a real incident is a guess dressed up as a document. We've watched engineers hit step three of a "verified" runbook and discover the referenced script no longer exists, or a required permission was revoked in a security cleanup months prior. The only way to know a runbook works is to run it, deliberately, when nothing is on fire. Game days that force a team to execute the actual runbook against staging, not talk through it in a meeting, catch this reliably. Anything that hasn't been executed in the last two quarters gets flagged stale regardless of how recently it was edited, because editing a document and rehearsing a procedure test different things.

Write for the person who can't think straight

The engineer opening a runbook mid-incident is not the calm, well-rested person who wrote it. They're paged, context-switched out of sleep or a weekend, reading under time pressure with an incident channel filling up around them. A runbook full of prose paragraphs and conditional clauses asks that person to parse and decide when they should only have to execute. The ones that actually get followed are numbered, imperative, and branch explicitly: check this metric, if it's above X do this, if below do that. No step should require the reader to hold more than one fact in their head to act on it. And every runbook should state its assumptions at the top, in plain terms, because the fastest way to lose a responder's trust is a step three that silently assumes a precondition step one never mentioned.

What we build instead

On engagements where we own reliability, runbook review is part of the same cadence as the on-call rotation review, not a separate documentation initiative that competes for time and loses. A runbook is treated as production code: it has an owner, it's versioned alongside the service it covers, and it gets exercised on a schedule, not just written once with good intentions. That's the only way we've seen the gap close between "we have a runbook" and "the runbook is the reason the incident closed in twenty minutes instead of two hours."

The uncomfortable test for any runbook sitting in your wiki right now: would you bet your own 3am on it? If the honest answer is no, it isn't a runbook. It's a document that happens to describe an incident procedure, which is a very different thing.