Serving with FastAPI

A prediction over HTTP.

⏱ 2 min read▶ Video at 40:43🎓 Free

Watch this lesson in the MLOps courseWatch on YouTube ↗Full MLOps course

A model in a file helps nobody. Applications need to ask it questions, and get answers, over the network. The usual way is a small web service. You send customer data, it replies with a prediction. In this lesson, we build one with Fast A P I, a modern, fast Python framework.

Wrap the model in a web service
Send data, receive a prediction
Validate inputs automatically

The prediction service

Here is the whole service. It loads the model once, when the program starts, not for every request. The customer class describes the expected input, four fields, with their types. Fast A P I will check every request against it. A health endpoint returns, status ok, so that Kubernetes can check the service is alive. And the predict endpoint receives a customer, turns it into a table row, asks the model for the churn probability, and returns it, with the model version. Short, clear, and safe.

app.pypython

import joblib
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel

model = joblib.load("model.joblib")
app = FastAPI(title="Churn API")


class Customer(BaseModel):
    tenure: int
    charges: float
    calls: int
    contract: int


@app.get("/health")
def health():
    return {"status": "ok"}


@app.post("/predict")
def predict(customer: Customer):
    row = pd.DataFrame([customer.model_dump()])
    probability = float(model.predict_proba(row)[0][1])
    return {"churn_probability": round(probability, 3), "model_version": "1"}

Calling the API

Let's run it. The server starts, and we call the health endpoint, which answers ok. Now a prediction. We send a new customer, three months of tenure, a high bill, four support calls, and a monthly contract. The answer, a churn probability of point eight two, with the model version. Now a mistake. We send letters instead of a number, for tenure. The service does not crash. It answers with a clear validation error, naming the exact field. That is the benefit of declaring the input shape.

terminal

uvicorn app:app --port 8000
INFO:     Started server process [29773]
INFO:     Waiting for application startup.
curl -s localhost:8000/health
{"status":"ok"}
curl -s -X POST localhost:8000/predict -H 'Content-Type: application/json' -d '{"tenure":3,"charges":95,"calls":4,"contract":0}'
{"churn_probability":0.82,"model_version":"1"}
curl -s -X POST localhost:8000/predict -H 'Content-Type: application/json' -d '{"tenure":"abc","charges":95,"calls":4,"contract":0}' | cut -c1-130
{"detail":[{"type":"int_parsing","loc":["body","tenure"],"msg":"Input should be a valid integer, unable to parse string as an inte

File used: churn.py

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

File used: train_model.py

import joblib
from sklearn.linear_model import LogisticRegression
from churn import make_data

df = make_data(3000, seed=1)
model = LogisticRegression(max_iter=500).fit(df.drop(columns="churn"), df["churn"])
joblib.dump(model, "model.joblib")

File used: app.py

import joblib
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel

model = joblib.load("model.joblib")
app = FastAPI(title="Churn API")


class Customer(BaseModel):
    tenure: int
    charges: float
    calls: int
    contract: int


@app.get("/health")
def health():
    return {"status": "ok"}


@app.post("/predict")
def predict(customer: Customer):
    row = pd.DataFrame([customer.model_dump()])
    probability = float(model.predict_proba(row)[0][1])
    return {"churn_probability": round(probability, 3), "model_version": "1"}

File used: requirements.txt

fastapi==0.162.0
joblib==1.6.0
numpy==2.5.4
pandas==3.1.0
pydantic==2.14.2
scikit-learn==1.9.1
uvicorn==0.54.0

File used: Dockerfile

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py churn.py model.joblib ./
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

What a good model API has

A production model service needs more than a predict endpoint. A health check, and a readiness check. A version in every response, so you can trace answers. Input validation, with clear errors. Logging of requests, and predictions, which will feed monitoring later. Timeouts, so one slow call cannot block everything. And authentication, so that only allowed clients can call it. Fast A P I gives you most of the plumbing, and you add the rest.

Health and readinessfor Kubernetes probes

Version in responsestrace every answer

Validation and loggingclear errors, useful records

Timeouts and authprotect the service

Summary

Excellent. A Fast A P I service loads the model once, validates inputs, and answers predictions over H T T P. Add health checks, versions, and logging. Next, the three big patterns for serving, batch, online, and streaming, and how to choose.

Key takeaways

Load the model once at startup
Declare the input shape
Health, version, logging

Next up: Serving Patterns — batch, online, and streaming

❮ Containers for ML Serving Patterns ❯