Capstone: The Gate

Step three. A trained model is not yet a production model. First it must pass tests. Then it must prove that it is better than the current champion, before it takes its place. This gate is what lets a team retrain often, without fear. Let's read the gate, and then try it, with a good model, and a worse one.

• Tests first
• Better than the champion
• Reject worse models automatically

The promotion gate

Here is the promotion script. It reads the candidate's F one score, and loads the model. Then it asks the registry for the current champion, and its recorded score. If there is no champion yet, the candidate only has to clear a floor of point three. If the candidate beats that score, the script logs the model, registers a new version, records its score as a tag, and moves the champion alias to it. Otherwise, it prints that the candidate was rejected. The decision is automatic, and recorded.

import json
import joblib
import mlflow
from mlflow import MlflowClient

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
NAME = "churn-model"

candidate_f1 = json.load(open("metrics.json"))["f1"]
model = joblib.load("models/model.joblib")

try:
    champion = client.get_model_version_by_alias(NAME, "champion")
    champion_f1 = float(champion.tags["f1"])
except Exception:
    champion_f1 = 0.30

if candidate_f1 > champion_f1:
    with mlflow.start_run():
        info = mlflow.sklearn.log_model(model, name="model", serialization_format="cloudpickle")
    version = mlflow.register_model(info.model_uri, NAME)
    client.set_model_version_tag(NAME, version.version, "f1", str(candidate_f1))
    client.set_registered_model_alias(NAME, "champion", version.version)
    print(f"PROMOTED version {version.version}: f1 {candidate_f1} beats {champion_f1}")
else:
    print(f"REJECTED: f1 {candidate_f1} does not beat champion {champion_f1}")

Tests, then promotion

Let's try it. First, the tests. Both pass, the model clears the quality floor, and returns valid probabilities. Now we run the gate. There is no champion yet, so the model is promoted, as version one. We run it again, with the same model. This time, it is rejected, because it does not beat the champion, it only equals it. Now we try a variation, a deeper forest, as an experiment. Its score is lower, so the gate rejects it, as well. The champion stays in place, and no human had to decide.

pytest -q -p no:cacheprovider -W ignore
..                                                                       [100%]
2 passed in 0.92s
python src/promote.py 2>&1 | grep -E 'PROMOTED|REJECTED'
PROMOTED version 1: f1 0.389 beats 0.3
python src/promote.py 2>&1 | grep -E 'PROMOTED|REJECTED'
REJECTED: f1 0.389 does not beat champion 0.389
git add -A > /dev/null && git commit -qm 'pipeline v1'
dvc exp run -S train.max_depth=10 2>&1 | grep -E '^f1'
f1 0.355
python src/promote.py 2>&1 | grep -E 'PROMOTED|REJECTED'
REJECTED: f1 0.355 does not beat champion 0.389

Summary