Real-Time Budgets
Every millisecond counts.
Twelve apples pass the camera every second. So you've got eighty three milliseconds for each one. Capture, inference, and firing the air jet all have to fit. Miss it, and a bad apple goes straight through. In real-time vision, speed isn't a nice extra. It's a hard limit.
- Twelve apples a second
- Eighty three milliseconds each
- Speed is a hard limit
A latency budget
Write down a simple budget. Twelve apples per second gives us eighty three milliseconds. Capture takes eight, the model takes twenty five, and sending the signal takes ten. Run it. We use forty three, and we've got forty spare. Always leave a margin, because the slowest frame matters more than the average. If a frame sometimes takes a hundred milliseconds, then an apple gets missed.
apples_per_second = 12
frame_budget_ms = 1000 / apples_per_second
capture, model, send = 8, 25, 10
used = capture + model + send
print(f"Budget per apple: {frame_budget_ms:.0f} ms")
print(f"Used: {used} ms, spare: {frame_budget_ms - used:.0f} ms")Budget per apple: 83 ms Used: 43 ms, spare: 40 ms
Batching
Models run faster per image when they're given several at once. We measure single images, and batches. Run it. Batches of two hundred and fifty six handle about half as many again per second. That's good for overnight jobs. But be careful. On a live belt, a batch means waiting for images to arrive, so batching adds delay. High throughput and low latency are different goals. Pick the one that your problem actually needs.
import time
import torch
from patch import make, net
model = net().eval()
X, _ = make(256, seed=1)
with torch.no_grad():
for batch in [1, 16, 256]:
start = time.time()
for _ in range(256 // batch):
model(X[:batch])
rate = (256 // batch) * batch / (time.time() - start)
print(f"Batch {batch:>3}: {rate:,.0f} images a second")Batch 1: 4,208 images a second Batch 16: 3,322 images a second Batch 256: 7,317 images a second
Also uses patch.py
import numpy as np
import cv2
import torch
def make(n, seed=0, light=1.0, bruise_rate=0.5):
"""Simulated apple photos. Label 1 means bruised."""
rng = np.random.default_rng(seed)
X, y = [], []
for _ in range(n):
img = np.full((48, 48, 3), (70, 75, 80), np.uint8)
c = tuple(int(v) for v in rng.integers(19, 30, 2))
r = int(rng.integers(15, 20))
cv2.circle(img, c, r, (35, 40, int(rng.integers(150, 225))), -1, cv2.LINE_AA)
bad = rng.random() < bruise_rate
if bad:
at = (c[0] + int(rng.integers(-8, 9)), c[1] + int(rng.integers(-8, 9)))
cv2.circle(img, at, int(rng.integers(3, 6)), (25, 45, 70), -1, cv2.LINE_AA)
img = img * light + rng.normal(0, 8, img.shape)
X.append(img.clip(0, 255) / 255)
y.append(int(bad))
X = torch.tensor(np.array(X), dtype=torch.float32).permute(0, 3, 1, 2)
return X, torch.tensor(y)
def net():
return torch.nn.Sequential(
torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Flatten(), torch.nn.Linear(32 * 12 * 12, 2))
def train(model, X, y, epochs=8):
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
for _ in range(epochs):
for i in torch.randperm(len(X)).split(64):
loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
opt.zero_grad(); loss.backward(); opt.step()
return model
def accuracy(model, X, y):
return (model(X).argmax(1) == y).float().mean().item()Where to run it
Where should the model run? In the cloud, you've got huge compute, but you pay for the round trip, and you need a network. An edge box next to the camera gives you low delay, and it works offline. An on camera chip is cheapest per unit, but only suits small models. Most factories choose the edge.
| Where | Best for |
|---|---|
| Cloud | Heavy batch jobs |
| Edge box | Low delay, no internet needed |
| On the camera | Small models, low cost |
| Hybrid | Edge now, cloud review later |
Summary
Budget every millisecond, and keep a margin. Next, let's look up from the apples, and see where these skills take you.
Key takeaways- Write a latency budget
- Throughput isn't latency
- Edge suits factories