Model Dashboards

When something seems wrong with a model, people ask, can you show me? A good dashboard answers that in seconds. It puts the few numbers that matter on one screen, with thresholds drawn in, so that anyone can tell, at a glance, whether the model is healthy. Let's build one, with real code, and see what belongs on it.

• Can you show me?
• A few numbers, one screen
• Thresholds drawn in

A health dashboard in code

Here is a small dashboard, drawn with code. Two panels share a time axis, the weeks. The top panel plots the model's accuracy, with the alert threshold as a dashed line. The bottom panel shows the drift score of the charges feature, as bars, coloured green while it is low, and orange once it passes the limit. The picture tells the story at once. In week four, drift starts to climb. Accuracy follows, and, from week five, it falls below the threshold. Drift gave the early warning, and accuracy confirmed the damage.

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

weeks = list(range(1, 9))
accuracy = [0.88, 0.87, 0.87, 0.81, 0.73, 0.65, 0.63, 0.62]
psi = [0.01, 0.02, 0.02, 0.18, 0.42, 0.66, 0.70, 0.71]

fig, (top, bottom) = plt.subplots(2, 1, figsize=(9, 6), sharex=True)
top.plot(weeks, accuracy, marker="o", color="#19c3a6")
top.axhline(0.80, color="#ff5c8a", linestyle="--", label="alert threshold")
top.set_ylabel("accuracy")
top.set_title("Churn model: weekly health")
top.legend()
bottom.bar(weeks, psi, color=["#19c3a6" if v < 0.25 else "#ffb86b" for v in psi])
bottom.axhline(0.25, color="#ff5c8a", linestyle="--")
bottom.set_ylabel("PSI of charges")
bottom.set_xlabel("week")
plt.tight_layout()
plt.savefig("dashboard.png", dpi=110)
print("saved dashboard.png")

saved dashboard.png

What belongs on the dashboard

What should such a dashboard contain? The service layer, request rate, errors, and latency percentiles. The data layer, drift scores for the key features, and the share of missing values. The model layer, prediction distribution, and, when labels arrive, accuracy or F one over time. The business layer, the outcome the model exists to improve. And the freshness of data, and of the model, in days. Keep it to one screen. A dashboard that needs scrolling, is a report, not a dashboard.

Summary