Serving and Monitoring
Keeping vision systems healthy.
Vision systems don't crash when they go wrong. They just slowly get worse. A lens gets dusty. The sun moves. Autumn turns to winter, and the light changes. Six months after launch, your accuracy has fallen, and nobody's noticed.
- Vision systems fail quietly
- Dust, sun, and seasons
- Nobody notices
An image API
For systems that aren't on the edge, a simple API is enough. A client uploads a photo, and the server replies with a grade. Here's a small one, with a placeholder rule, so that you can see the shape of it. Run it. Status two hundred, and the result as JSON. Swap in your real model, and you've got a service.
import io
import numpy as np
from fastapi import FastAPI, UploadFile
from fastapi.testclient import TestClient
from PIL import Image
app = FastAPI()
@app.post("/grade")
async def grade(file: UploadFile):
image = np.array(Image.open(io.BytesIO(await file.read())).convert("RGB"))
red = image[..., 0].mean()
return {"width": image.shape[1], "red_level": round(float(red), 1),
"grade": "A" if red > 120 else "B"}
buf = io.BytesIO()
Image.fromarray(np.full((64, 64, 3), (200, 40, 40), np.uint8)).save(buf, "JPEG")
reply = TestClient(app).post("/grade", files={"file": ("apple.jpg", buf.getvalue())})
print(reply.status_code, reply.json())200 {'width': 64, 'red_level': 200.0, 'grade': 'A'}Catching drift
The cheapest monitor watches simple image statistics, like brightness. Here we compare live brightness with what the model saw in training. In winter, the barn gets much darker. Run it. Week one and week eight look fine. Winter raises an alert, before the accuracy falls too far. You don't need labels for this. It's cheap, and it catches many real failures.
import numpy as np
rng = np.random.default_rng(0)
train_brightness = rng.normal(120, 12, 5000)
for week, mean in [("Week 1", 121), ("Week 8", 118), ("Winter", 86)]:
live = rng.normal(mean, 14, 500)
shift = abs(live.mean() - train_brightness.mean()) / train_brightness.std()
print(f"{week:<7} brightness {live.mean():5.1f} shift {shift:.2f} "
f"{'ALERT' if shift > 1 else 'ok'}")Week 1 brightness 120.7 shift 0.06 ok Week 8 brightness 118.7 shift 0.10 ok Winter brightness 86.0 shift 2.84 ALERT
A learning loop
The best systems keep improving. Send low confidence images to a person. Also audit a small random sample, to catch mistakes that the model is confident about. Those new labels feed the next training run. Run it. A manageable number of images each week. This loop is how good systems get great. Use a tool like Label Studio to run it.
import numpy as np
rng = np.random.default_rng(0)
confidence = rng.beta(6, 2, 10000)
review = confidence < 0.6
sampled = rng.random(10000) < 0.01
print(f"Sent to human review: {review.mean():.1%}")
print(f"Random audit sample: {sampled.mean():.1%}")
print("Images per week to label:", int((review | sampled).sum()))Sent to human review: 16.0% Random audit sample: 1.1% Images per week to label: 1688
Summary
Watch your inputs, keep an API simple, and feed the hard cases back into training. You've now covered the full life of a vision system. Next, how to get far more from the data that you've already got.
Key takeaways- Vision fails quietly
- Monitor image statistics
- Hard cases become training data