Serving a Network

Aria has grown. She can see, read, write, and imagine. But she still lives in a notebook on your laptop. It's time to give her an address on the internet, so that real people can use her.

• Aria can do a lot now
• But she lives in a notebook
• Give her an address

A model behind an API

Fast A P I makes this easy. We load the model when the server starts, one time only. We define the request shape. In the predict function, we use inference mode, run the model, and return chances. Then we call it, using a test client. Run it. A status of two hundred, and three chances. That's a real service. Run it with uvicorn, and it's live.

import io
import torch
from torch import nn
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pydantic import BaseModel

model = nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 3))
model.eval()
app = FastAPI()

class Item(BaseModel):
    features: list[float]

@app.post("/predict")
def predict(item: Item):
    with torch.inference_mode():
        scores = model(torch.tensor([item.features]))
    return {"chances": scores.softmax(1)[0].round(decimals=3).tolist()}

reply = TestClient(app).post("/predict", json={"features": [0.1, 0.5, 0.2, 0.9]})
print(reply.status_code, reply.json())

200 {'chances': [0.421999990940094, 0.3149999976158142, 0.2630000114440918]}

What real serving needs

Production needs a little more. Batching groups many requests together, because G P Us work best that way. Streaming sends tokens as they're generated, which is how chatbots feel fast. Caching avoids repeating work. And you should watch latency, cost per request, and quality. Tools such as vLLM and T G I solve these for language models. Our M L Ops course covers this in depth.

Summary