What to Monitor

Deploying a model is not the finish line, it is the starting line. From the first minute in production, you need eyes on it. But what should you watch? A server can be perfectly healthy while the model inside it gives terrible answers. In this lesson, we organise monitoring into four layers.

• Healthy server, bad model
• Four layers to watch
• Watch from the first minute

Four layers of monitoring

Think of monitoring as four layers. At the bottom, infrastructure. Is the machine alive, with enough memory and processor? Next, the service. How many requests, how fast, how many errors? Above that, the data. Are the inputs still similar to what the model learned from? And at the top, the model's quality. Are the predictions still right? Most teams monitor the lower layers well, and the upper layers poorly. But the upper layers are exactly where machine learning systems fail silently.

1. InfrastructureCPU, memory, disk, restarts
2. Servicerequests, latency, errors
3. Datainput distributions, missing values
4. Model qualityaccuracy, business results

Metrics a service exposes

The service layer is easy to measure, with Prometheus. Here we create two instruments. A counter, which only goes up, counts predictions, labelled by outcome. A histogram records how long each prediction took, in buckets. We simulate two hundred requests, and then print the metrics, in the text format that Prometheus reads. We see how many predictions said churn, and how many said stay. And the histogram shows that most requests finished within fifty milliseconds, almost all within a tenth of a second.

import random
from prometheus_client import Counter, Histogram, generate_latest, CollectorRegistry

registry = CollectorRegistry()
requests = Counter("predictions_total", "Predictions served", ["outcome"], registry=registry)
latency = Histogram("prediction_seconds", "Time to predict", buckets=(0.01, 0.05, 0.1, 0.5), registry=registry)

random.seed(0)
for _ in range(200):
    seconds = random.expovariate(1 / 0.03)
    latency.observe(seconds)
    requests.labels("churn" if random.random() < 0.15 else "stay").inc()

for line in generate_latest(registry).decode().splitlines():
    if line.startswith(("predictions_total", "prediction_seconds_bucket", "prediction_seconds_count")):
        print(line)

predictions_total{outcome="stay"} 173.0
predictions_total{outcome="churn"} 27.0
prediction_seconds_bucket{le="0.01"} 50.0
prediction_seconds_bucket{le="0.05"} 156.0
prediction_seconds_bucket{le="0.1"} 195.0
prediction_seconds_bucket{le="0.5"} 200.0
prediction_seconds_bucket{le="+Inf"} 200.0
prediction_seconds_count 200.0

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Prediction monitoring

Besides the usual service numbers, log the predictions themselves. Record the share of each predicted class over time. A sudden change in that share is an early warning. Record the model's confidence. Record the model version, with every prediction, so that you can trace any answer. And store the inputs, within privacy rules, so that you can analyse them later, and replay them against new models. These records are the raw material for everything in the next lessons.

Summary