Service Level Objectives

How reliable should a model service be? Perfect is impossible, and chasing it is ruinously expensive. The professional answer is to choose a target, measure against it, and spend the remaining slack on improving the product. These targets are called service level objectives. Let's see how they work.

• Perfect is impossible
• Choose a target on purpose
• Spend the slack on progress

Indicators, objectives, agreements

Three terms are easy to mix up. A service level indicator, S L I, is a measurement, like the share of requests that succeed, or finish within a hundred and fifty milliseconds. A service level objective, S L O, is the target for that measurement, for example, ninety nine point nine percent of requests succeed, over thirty days. And a service level agreement, S L A, is a contract with customers, with penalties if the target is missed. Teams set internal objectives, that are stricter than any agreement they promise.

Measuring against the target

Here is an S L O check, on simulated traffic. One hundred thousand requests, each with a latency, and a chance of failing. We compute the availability, and compare it with the ninety nine point nine percent target. We compute the share of fast requests, below one hundred and fifty milliseconds. Then comes the useful idea, the error budget. A target of ninety nine point nine percent allows one request in a thousand to fail. Over a hundred thousand requests, that is one hundred failures. We count how many we have used. Here, sixty eight percent. While budget remains, you can ship changes boldly. When it runs out, you freeze releases, and fix reliability.

import numpy as np

rng = np.random.default_rng(1)
n = 100_000
latency_ms = rng.lognormal(mean=3.4, sigma=0.5, size=n)
failed = rng.random(n) < 0.0007
slow = latency_ms > 150

print(f"availability   {1 - failed.mean():.3%}   target 99.9%")
print(f"fast requests  {1 - slow.mean():.2%}   target 99% under 150 ms")

budget = 0.001 * n
used = failed.sum()
print(f"error budget   {budget:.0f} failures allowed, {used} used ({used / budget:.0%})")
print("verdict       ", "healthy" if used < budget else "freeze releases, fix reliability")

availability   99.932%   target 99.9%
fast requests  99.91%   target 99% under 150 ms
error budget   100 failures allowed, 68 used (68%)
verdict        healthy

SLOs for ML systems

Machine learning systems need objectives beyond the ordinary ones. Availability and latency of the service, as usual. Freshness, how old is the data, and the model. Quality, for example, accuracy over the last month must stay above a floor. And coverage, the share of requests that get a prediction, instead of a fallback. Choose a few that users truly feel, write them down, and review them quarterly. An objective nobody measures, is only a wish.

Summary