How to Actually Decide & Agree on an SLO?

Someone asked me in an interview what p99 latency I’d target for a product that had gotten slow. I almost gave them a number.

I didn’t, because the number was never the hard part. Any engineer who’s been around knows to use percentiles, knows the SLA should sit looser than the internal target, knows what an error budget is. That’s the easy 20%. The hard 80% is two questions that don’t belong to engineering alone: how much is slow actually costing us, and how far is it worth pushing to fix it. Those get answered in a room, with a product lead who wants the number aggressive for the roadmap, a sales lead who wants a promise they can say on a call, and two engineers who want enough headroom to sleep.

So this is about the parts nobody really trains you for. Not the percentiles. How you work out what slow is worth, how much to optimize before the next 100ms costs more than it returns, and how people who want different things land on one number together and keep it honest once the product moves on. Decide it alone at an engineer’s desk and it won’t survive contact with the first roadmap crunch.

The one minute version of the SLI, SLO, SLA & Error Budget

So we’re on the same page, fast. The SLI is the thing you measure, like p95 latency on the booking call. The SLO is the internal target you hold the team to. The SLA is the weaker promise you make to a customer, with a penalty attached. The error budget is whatever’s left over from the SLO: if your target is 99.9%, then 0.1% is the unreliability you’re allowed to spend.

That’s the whole vocabulary, and it’s not where teams go wrong. They go wrong turning that vocabulary into a commitment. Here’s what makes that go well or badly.

Walk in with a baseline and a cost curve

The fastest way to lose the room is to open with “so, what should our latency target be?” Now everyone’s guessing, and in a guessing match the most confident voice wins, which is rarely the most correct one.

Bring two things instead. Where you are today: p95 is 600ms, here’s the graph. And what each improvement costs: getting to 300ms is a caching project, maybe two sprints; getting to 150ms means reworking the availability queries, closer to a quarter. Now the discussion has edges. People are choosing between real options with real prices, not pulling numbers out of the air. Half the bad SLO conversations I’ve sat in went sideways for the plain reason that nobody brought the cost of the thing they were arguing about.

Where the first number actually comes from

The baseline tells you where you are. It doesn’t tell you where the bar should be, and that’s the question people freeze on, because a target pulled from nowhere falls apart the second someone asks “why 200 and not 300?”

The most useful way I’ve answered that is to run it backwards. Instead of asking how fast we should be, measure what slow costs. You do it with a negative A/B test: you deliberately add latency to a small slice of traffic and watch what happens to the numbers the business actually cares about.

The word “deliberately” makes people nervous, so the design matters more than the idea. We didn’t just switch it on. We picked the market mix first, because 100ms doesn’t cost the same everywhere. Network conditions, devices, and whether the user has an easy alternative all move the answer, and a global average would have buried that. We held it to 1% of traffic, time-boxed, with a kill-switch, because you’re spending real money to learn this and you want the smallest bill that still gives a clean signal. Then we read the delta: how much did each added 100ms move conversion, or whatever metric pays the bills. That gave us the thing you actually need, a rough curve of business cost against milliseconds. And from there the target stops being a preference. It’s the point where the next 100ms of speed costs more to build than the revenue it buys back. That’s also how you connect effort to gain honestly, because now “make it faster” has a price tag and a payoff sitting next to each other.

That method has sharp edges, and I’d name them before leadership does. You’re intentionally degrading real users to learn something, so it’s off the table for anything safety- or trust-critical, and it needs explicit sign-off, a hard cap, and a kill-switch you’ve actually tested. At 1% you need enough traffic and enough time to see a small effect through the noise, so a low-traffic product can’t run this cleanly, and forcing it gives you a confident number built on nothing. And you only learn about the range you tested. Add 200ms and you’ve measured the cost of 200ms, not 800ms, and the curve isn’t a straight line, because there are perception cliffs where it drops off a ledge. Slowing down isn’t a perfect mirror of speeding up either. So what you get is a strong anchor for the conversation, not a law.

Most teams can’t or shouldn’t run that experiment, and there are cheaper ways to propose a first number that still beat guessing. The safest is to start where you are and refuse to regress: your first SLO can simply be “as good as today, and we don’t backslide.” It costs nothing to justify and buys you time to gather data for a sharper number. You can anchor to how people perceive delay, since under about 100ms feels instant, around a second is where attention starts to wander, and past a few seconds people assume it’s broken. Or start loose on purpose: propose a number you already hit almost all the time, watch the error budget for a quarter, and tighten once you know what the next step costs. It’s far easier to tighten an SLO the team is beating than to walk back one you set too aggressively and keep missing.

Translate every number before anyone opens their mouth

“p99” means nothing to a product manager and less to someone in sales. If you put a percentile on a slide and wait for their opinion, you’ll get silence or a nod, and a nod isn’t agreement. It’s the sound of someone who checked out.

Your job is to show up having already translated. “A p99 of four seconds means one booking in a hundred makes a user wait long enough to assume it’s broken and refresh.” Now the product lead can weigh in, because you’ve handed them the thing in a currency they own: user trust. Sales needs a different translation again, because they don’t feel availability percentages, they feel the demo stalling in front of a prospect. Same number, three translations, all done before the meeting so everyone can hold an opinion worth having.

Make them feel the cost, or they’ll always want it tighter

This one took me too long to learn. If tightening the number is free to the person asking for it, they’ll always ask for it tighter. Why wouldn’t they? Faster is better, more nines are better, and from where they sit there’s no reason to stop.

So you move the cost onto their side of the table, out loud. Not “that’s expensive,” because expensive is abstract. It’s “we can hit 150ms, and the price is the two-sprint feature we don’t ship this quarter to do it. Your call.” The moment the tradeoff is theirs to make instead of only yours to absorb, the conversation gets honest in a hurry. People who wanted five nines a minute ago suddenly find that three and a half is plenty. Nothing focuses a request like a bill attached to it.

Don’t smooth the disagreement, that’s the whole meeting

There’s a real tension in the room, and the instinct is to paper over it. Resist that. Engineering wants headroom. Product wants the aggressive number that makes the roadmap look good. Sales wants a promise they can make on a call without checking. Those aren’t personality clashes to be managed away. They’re three honest views of the same system, and the SLO is where they get resolved in the open instead of festering into “engineering is slow” and “product is reckless.”

This is where the error budget stops being a definition and becomes a tool. When product and engineering deadlock on features versus stability, the budget is the neutral thing that breaks the tie. Budget healthy, we ship. Budget spent, we stabilize. Nobody has to win on force of personality, because there’s a number both sides agreed to ahead of time. That’s the quiet reason error budgets matter, and it has nothing to do with reliability math. They turn a recurring fight into a rule.

Write down who owns the “then what”

Most SLO meetings end the second everyone agrees on the number, which is exactly when they should keep going for five more minutes. A target with no consequence is a wish. The question that actually matters is: when we breach this, what happens, and who decides?

Does red mean the team stops feature work until it’s back? Does it page someone? Does someone call the customer before the customer calls us? Name it, and name the person who owns that call. An SLO with a “then what” is a commitment. An SLO without one is a chart you’ll quietly stop opening by the second month.

The number has an expiry date

A target that was right in March is often wrong by September, and treating an SLO as permanent is how it quietly stops meaning anything. Put a review on the calendar, once a quarter is usually enough, and let the error budget tell you what to do when you get there.

If you’ve been sitting far under budget for months, barely touching the unreliability you’re allowed, that isn’t a gold star. It means you’re buying more reliability than the business is asking for, and that surplus is effort you could move to features. Loosen the target and redirect the team. Over-reliability is spare capacity nobody’s spending, and the review is where you find it.

If you keep blowing the budget, the number is either wrong or under-resourced, and the honest move is to say so early instead of missing it quietly every month. Tighten it, resource it, or renegotiate the promise. All three are fine. Pretending it still holds is not. That same review is where you catch the flow that used to be a side feature and is now the thing customers show up for, or the surface that didn’t exist last quarter. The set of things worth an SLO changes as the product does, and if your targets never move, they’ve stopped describing the product you actually have.

So when someone asks what number you’ll hit, the strong answer isn’t a fast number. It’s “let me measure what the slow version costs us, get the right people in a room with that data, and come back with one we’ve all signed off on, and we’ll revisit it in a quarter.” I think that’s what the interview question was really testing. Not whether I knew what a p99 was. Whether I knew the number is the easy part, and the judgment around it is the job.