Serving Patterns

Not every model needs to answer in milliseconds. Some run overnight, over millions of rows. Others answer a click, instantly. Choosing the right pattern saves money, and prevents a lot of pain. In this lesson, we compare the three, and see how to measure speed.

• Three ways to serve
• Match the pattern to the need
• Measure latency properly

Batch, online, streaming

Here are the three patterns. In batch, a job runs on a schedule, scores a large table, and writes the results to a database. It is cheap, simple, and perfect when predictions can be a few hours old, like a weekly list of customers at risk. In online serving, a service answers each request, in milliseconds, which is needed for things like fraud checks, and recommendations. And in streaming, a processor scores events, as they flow through a message queue, for near real time results. Start with batch, if you can. Move to online only when freshness truly matters.

Choosing the right pattern

How do you choose? Ask how fresh a prediction must be. If a weekly update is fine, batch is the cheapest, and the easiest to operate. Ask how many requests arrive. Millions of rows, once, favour batch. Thousands of requests, per second, need online serving, with scaling. Ask what the features need. If they depend on the last few seconds, streaming, or online is needed. More freshness always costs more complexity, so buy only what you need.

Measuring latency

Speed claims need numbers. Here we send two hundred predictions, one by one, to the model, and measure each. We print the median, which is the typical request, and the ninety ninth percentile, the slow tail, which is what unhappy users feel. Always look at percentiles, never only the average. And measure the whole service, not only the model, because networking, and validation add time too.

import time
import numpy as np
from sklearn.linear_model import LogisticRegression
from churn import make_data

df = make_data(3000, seed=1)
model = LogisticRegression(max_iter=500).fit(df.drop(columns='churn'), df['churn'])
row = df.drop(columns='churn').head(1)

times = []
for _ in range(200):
    start = time.perf_counter()
    model.predict_proba(row)
    times.append((time.perf_counter() - start) * 1000)

print('median ms:', round(float(np.percentile(times, 50)), 2))
print('p99 ms   :', round(float(np.percentile(times, 99)), 2))

median ms: 0.23
p99 ms   : 0.37

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Summary