Capstone: Operate

Step six, and the most important. The model is live, and time passes. A week later, new data arrives, and the world has changed. Our job is to notice, automatically, to retrain, on fresh data, and let the gate decide. This closes the loop of the lifecycle, and it is the heart of M L Ops.

• A week passes
• Notice the change
• Retrain, and let the gate decide

Detect drift, retrain, promote

This is the weekly check. We generate this week's data, and run the drift detector against the reference data. Tenure looks the same, but charges and calls have drifted, and the script exits with a failure code, which a scheduler treats as, retrain. So we retrain, as an experiment, with fresh, shifted data, by changing two parameters. The pipeline reruns the prepare, and train stages, and reports a new F one score. Then the gate compares it with the champion, and promotes it, as version two. In a real system, you would compare both models on the same fresh test set, to be fair. The loop is closed, automatically.

python src/incoming.py
wrote data/incoming.csv (this week's data)
python src/drift.py; echo "exit code: $?"
tenure   p-value 0.7094  ok
charges  p-value 0.0000  DRIFT
calls    p-value 0.0000  DRIFT
drift in ['charges', 'calls'] -> retrain
exit code: 1
dvc exp run -S prepare.seed=3 -S prepare.shifted=true 2>&1 | grep -E '^prepared|^f1'
prepared data/churn.csv, seed 3 shifted
f1 0.512
python src/promote.py 2>&1 | grep -E 'PROMOTED|REJECTED'
PROMOTED version 2: f1 0.512 beats 0.389
python -c "import mlflow; mlflow.set_tracking_uri('sqlite:///mlflow.db'); print('champion is version', mlflow.MlflowClient().get_model_version_by_alias('churn-model', 'champion').version)" 2>/dev/null
champion is version 2

What we built

Look back at what you built. A repository, with a clear layout. Versioned, validated data. A reproducible pipeline, with tracked experiments. Tests, and a registry with a quality gate. A container, an A P I, and Kubernetes manifests. A delivery pipeline. And a monitoring loop, that detects drift, retrains, and promotes. Every stage of the life cycle is there, and every stage is automated, and traceable. That is M L Ops, and you have now built it.

Summary