Building a Reliability Culture That Doesn't Depend on Heroics

There's a version of "we have on-call" that means nothing more than a rotation schedule and a paging tool. It looks like reliability engineering from the outside, right up until the first serious incident, when it turns out the person holding the pager doesn't know the system well enough to do anything except escalate and wait. Real reliability culture is built long before the first page fires, and it's built into decisions that have nothing to do with incident response tooling.

Ownership has to match who's paged. The single most reliable predictor of whether an on-call rotation actually works is whether the people on it wrote the code they're being paged for. When the on-call engineer is disconnected from the build — a support contractor, an offshore rotation, a team that inherited the system after the builders moved on — every incident starts with archaeology instead of diagnosis. On our engagements, the engineers on the rotation are the ones who designed the data model and wrote the integration code, which means a 2am page starts with "I know exactly what that error means" instead of "let me go read the code."

Alerts have to be actionable or they get deleted, not tolerated. Alert fatigue is a reliability failure, not a minor annoyance — a team that's learned to ignore half its alerts has effectively turned off monitoring for those conditions. We set a hard rule during build: every alert that fires must correspond to an action a human can take, and if an alert has fired three times without anyone taking an action, it gets deleted or rewritten. This is uncomfortable in practice, because it means admitting that some carefully-configured threshold you set up in week two was noise, not signal.

The postmortem is for the system, not the person. Blameless postmortems get talked about often enough that the phrase has lost some of its meaning, but the practice underneath it is specific: every incident review asks "what about the system made this failure possible and how do we change the system," never "who made the mistake." This isn't a soft HR position — it's a hard engineering one. A culture that assigns blame to individuals teaches people to hide near-misses instead of reporting them, which means the next postmortem happens after the failure gets worse, not before.

Reliability work has to be budgeted, not squeezed into gaps. The single fastest way to erode a reliability culture is to treat resilience work — retry logic, graceful degradation, load testing, runbook writing — as something that happens only after feature work is done. It never gets done, because feature work is never done. We scope reliability work as a line item in the build plan with the same seriousness as feature scope, sized against the platform's actual criticality, not as a virtuous afterthought that competes for time against a launch deadline.

The runbook has to be written before the incident, and tested, not written after. A runbook drafted from memory during a live incident is worse than no runbook — it gives false confidence while someone improvises anyway. We write and dry-run runbooks for the failure modes we can anticipate during build, before launch, and we treat "we don't have a runbook for this" as a build-blocking gap for anything in the platform's critical path, not a post-launch nice-to-have.

The uncomfortable truth about most on-call setups is that they're designed to make the pager go off, not to make the person who answers it capable of fixing what's wrong. Fixing that isn't about better tooling. It's about making sure the person on the rotation is the person who actually understands the system, and making sure the system was built by people who had to think about 2am from day one.