Running on Devices
Making models small and fast.
At Orchard Vision, the camera is at the end of a dusty barn, with slow internet. Each apple must be graded in a fraction of a second. Sending every photo to the cloud isn't an option. The model has to run right there, on a small box next to the belt. That's called edge computing.
- A dusty barn, slow internet
- A fraction of a second per apple
- The model runs on the box
Exporting to ONNX
ONNX is a standard format for models, and it runs almost anywhere. We export a small network with one call, then load it into a fast runtime, with no PyTorch needed. Run it. A tiny file, and speed measured in milliseconds per image. Devices like the Jetson and the Raspberry Pi run files like this. Exporting is the usual first step toward any real deployment.
import os
import time
import numpy as np
import torch
from torch import nn
import onnxruntime as ort
net = nn.Sequential(nn.Conv2d(3, 16, 3, 2, 1), nn.ReLU(), nn.Conv2d(16, 32, 3, 2, 1),
nn.ReLU(), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 2)).eval()
x = torch.rand(1, 3, 128, 128)
torch.onnx.export(net, x, "model.onnx", input_names=["image"], output_names=["score"],
dynamo=False)
run = ort.InferenceSession("model.onnx")
a = np.random.rand(1, 3, 128, 128).astype(np.float32)
start = time.time()
for _ in range(100):
out = run.run(None, {"image": a})
print(f"File: {os.path.getsize('model.onnx') / 1000:.0f} KB")
print(f"Speed: {(time.time() - start) * 10:.2f} ms per image")File: 22 KB Speed: 0.10 ms per image
Budgeting your time
Everything starts with a time budget. A belt moves at a metre per second, and each apple stays in view for half a second. We compare three image sizes. Run it. Bigger images find small defects, but they cost much more time. Doubling the resolution makes it about four times slower. So choose the smallest size that still shows a bruise. This sum decides your hardware.
budget_ms = 100
resolutions = {320: 12, 640: 45, 1280: 180}
for size, ms in resolutions.items():
fps = 1000 / ms
verdict = "fits" if ms < budget_ms else "too slow"
print(f"{size:>4}px {ms:>3} ms {fps:>5.1f} fps {verdict}")
belt_speed_m_s, view_m = 1.0, 0.5
print("Time each apple stays in view:", view_m / belt_speed_m_s, "seconds")320px 12 ms 83.3 fps fits 640px 45 ms 22.2 fps fits 1280px 180 ms 5.6 fps too slow Time each apple stays in view: 0.5 seconds
The hidden costs
The model isn't the only cost. Resizing and compressing frames can take as long as inference. We time both. Run it. These steps take real time, on every frame. Compression shrinks the image a great deal, which matters if you're sending it across a slow link. Measure your whole pipeline, not just the model. The bottleneck is often somewhere else.
import cv2
import time
import numpy as np
frame = np.random.randint(0, 255, (1080, 1920, 3), np.uint8)
start = time.time()
for _ in range(100):
small = cv2.resize(frame, (640, 360))
print(f"Resize: {(time.time() - start) * 10:.1f} ms")
start = time.time()
for _ in range(100):
_, jpg = cv2.imencode(".jpg", frame)
print(f"JPEG encode: {(time.time() - start) * 10:.1f} ms")
print(f"Size: {frame.size / 1e6:.1f} MB -> {jpg.size / 1e6:.2f} MB")Resize: 0.2 ms JPEG encode: 10.3 ms Size: 6.2 MB -> 2.43 MB
Summary
Export, budget your time, and measure the full pipeline. Next, the other way to deploy, sending images to a server, and how to keep everything healthy after launch.
Key takeaways- ONNX runs anywhere
- Resolution costs time fast
- Measure the whole pipeline