SLOs and Error Budgets: The Practice, Not Just the Definition

Service level objectives have become standard vocabulary in enterprise engineering orgs, but there's a wide gap between organizations that have SLOs documented in a wiki and organizations where SLOs actually govern behavior. The tell is simple: ask what happened the last time a service burned through its error budget mid-quarter. If the answer is "nothing changed," the SLO is decoration.

Pick SLIs that reflect user experience, not infrastructure health

The most common mistake in early SLO programs is choosing service level indicators that are easy to measure rather than indicators that reflect what users actually experience. CPU utilization, pod restart count, and queue depth are operationally useful, but they're not SLIs — a service can have elevated CPU and a perfectly fine user experience, or normal CPU and a badly degraded one. Useful SLIs are almost always some combination of availability (the fraction of valid requests served successfully) and latency (the fraction of requests served under a threshold that reflects actual user tolerance, not an arbitrary round number).

For a checkout API, that means measuring the success rate and latency of the actual checkout completion endpoint under realistic request validity criteria — not "did the load balancer return a 200," which will look artificially healthy even when the application is failing every request with a well-formed 4xx or a fast, "successful" error response.

Error budgets only work if they have teeth

An error budget is the inverse of your SLO: a 99.9% availability target over 30 days allows roughly 43 minutes of budget to spend. The entire value of framing reliability this way is that the budget becomes a shared currency between product and engineering for making the feature-velocity-versus-stability tradeoff explicit, rather than litigating it informally every time something breaks. That only works if burning the budget has a real consequence, agreed to in advance: when a service's error budget is exhausted, feature deploys to that service pause and the team's priority shifts to reliability work until the trailing-window budget recovers.

We've implemented this as a literal gate in CI for enterprise clients — a deploy pipeline that checks the current error budget consumption via a query against the metrics backend before allowing a non-emergency deploy to proceed, with an explicit break-glass override that requires a second approver and gets logged for the next reliability review. The mechanism matters more than the policy, because a policy that relies on team discipline under deadline pressure erodes exactly when it's needed most.

Multi-window, multi-burn-rate alerting beats a single threshold

Alerting directly on "SLO violated" is both too slow (by the time the 30-day window shows a violation, the damage is long done) and too noisy (a genuine but brief spike triggers a page for something that will self-resolve within budget). The practice that's replaced naive threshold alerting is multi-window, multi-burn-rate alerting, popularized by Google's SRE workbook: alert with high urgency when the burn rate is fast enough to exhaust the entire budget in a few hours, and with lower urgency when a slower, sustained burn rate would exhaust it over a few days. This catches both the sharp outage and the slow-degradation pattern — a memory leak that costs a little availability every hour — without paging anyone for noise that resolves on its own.

The review cadence is where SLOs either compound or decay

Quarterly SLO reviews — not the postmortem-driven reactive review, a scheduled one — are where the actual governance happens: is the target still the right one, has user tolerance shifted, is the SLI still measuring what it claims to measure as the product has evolved underneath it. Services accumulate SLI drift the same way codebases accumulate technical debt; an SLO defined two product pivots ago is measuring the wrong thing with high precision. Treat the SLO definition itself as a piece of infrastructure that needs maintenance, reviewed on a fixed schedule against real usage data, not a document you write once during a reliability initiative and never revisit.