Batching and Hardware

Throughput, the number of predictions per second, decides how many servers you need, and so, how much you pay. Two things change it dramatically. How you group requests, which is batching, and what hardware you run on. Let's measure the first one, and then compare the second.

• Throughput sets the bill
• Batching groups requests
• Hardware changes the maths

One by one versus a batch

Here is a simple experiment. We train a forest of trees, and take five hundred customers. First, we predict them one at a time, in a loop, and time it. Then we predict all five hundred in a single batch call, and time that. Finally, we print the predictions per second for each, and the speed up. One by one, the model manages a few hundred predictions a second. In a batch, it manages tens of thousands. The model is the same. Only the grouping changed, and the speed up is hundreds of times.

import time
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from churn import make_data

df = make_data(3000, seed=1)
X, y = df.drop(columns="churn"), df["churn"]
model = RandomForestClassifier(n_estimators=100, max_depth=8, random_state=0, n_jobs=1).fit(X, y)
rows = X.head(500)

start = time.perf_counter()
for i in range(len(rows)):
    model.predict_proba(rows.iloc[[i]])
one_by_one = time.perf_counter() - start

start = time.perf_counter()
model.predict_proba(rows)
batched = time.perf_counter() - start

print(f"one by one : {len(rows) / one_by_one:8.0f} predictions per second")
print(f"one batch  : {len(rows) / batched:8.0f} predictions per second")
print(f"speed-up   : {one_by_one / batched:.0f}x")

one by one :      225 predictions per second
one batch  :    63390 predictions per second
speed-up   : 282x

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Dynamic batching in serving

Online services get one request at a time, so how can they batch? With dynamic batching. The server holds each request for a very short time, a few milliseconds, collects whatever else arrives in that window, and runs them together as one batch. Each user waits slightly longer, but the machine does far more work per second. The window and the maximum batch size are knobs. A bigger window raises throughput, and also latency. Tune them, with real traffic, against your latency target.

1. Requests arriveone at a time
2. Wait a few mscollect a small group
3. Run one batchone call for the group
4. Return each answerto its own caller

Choosing hardware

Hardware matters too. Ordinary processors are cheap, flexible, and great for classic models, like trees and linear models, and for small networks. Graphics processors, G P Us, excel at large neural networks, with massive parallel maths, and are essential for big language models, but cost much more per hour. Specialised chips, like T P Us, and cloud inference chips, can be cheaper per prediction, at scale, but are less flexible. Start on processors, move to graphics cards only when measurements show you need them.

Summary