Deployment Strategies

A new model passed all the tests. Now comes the scariest step, letting real customers meet it. If it is worse than the old one, real decisions suffer. Good teams never flip a switch and hope. They release in careful stages, and measure at each step. Let's learn the main strategies.

• Tests are not enough
• Real traffic is the real test
• Release in careful stages

Four ways to release a model

Here are four strategies, from safest to boldest. In shadow mode, the new model receives a copy of live requests, and predicts, but nobody sees its answers. You only compare. In a canary release, a small share of real traffic, say five percent, is served by the new model, and you watch closely. In an A B test, two groups of users see the two models, and you measure a business result, such as how many customers stayed. And in a full rollout, everyone switches, ideally with an easy way back.

1. Shadownew model predicts silently
2. Canarya small share of real traffic
3. A/B testtwo groups, measured outcome
4. Full rollouteveryone, with rollback ready

Three strategies, simulated

Here are three of them, simulated on real data. We have an old model, and a new random forest. Live traffic arrives, two thousand customers. In shadow mode, both models predict, but only the old answers are used. We print both scores, and compare them safely, with zero risk. In a canary release, ten percent of requests are served by the new model. We print the share, and the overall accuracy. In an A B test, we compare the two groups directly. Notice that the new group looks a little better, but with so few customers, that could be luck, so a real test needs enough data, and a statistical check.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from churn import make_data

train = make_data(3000, seed=1)
X, y = train.drop(columns="churn"), train["churn"]
old = LogisticRegression(max_iter=500).fit(X, y)
new = RandomForestClassifier(n_estimators=100, max_depth=6, random_state=0).fit(X, y)

live = make_data(2000, seed=5)
LX, ly = live.drop(columns="churn"), live["churn"]

old_pred, new_pred = old.predict(LX), new.predict(LX)
print("shadow: old", round(accuracy_score(ly, old_pred), 3), "| new, hidden from users", round(accuracy_score(ly, new_pred), 3))

use_new = np.random.default_rng(0).random(len(LX)) < 0.10
served = np.where(use_new, new_pred, old_pred)
print("canary:", round(use_new.mean() * 100), "% on the new model, overall accuracy", round(accuracy_score(ly, served), 3))

print("A/B: old group", round(accuracy_score(ly[~use_new], old_pred[~use_new]), 3), "| new group", round(accuracy_score(ly[use_new], new_pred[use_new]), 3))

shadow: old 0.868 | new, hidden from users 0.866
canary: 10 % on the new model, overall accuracy 0.867
A/B: old group 0.865 | new group 0.881

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Choosing and measuring

How do you choose? Use shadow mode when mistakes are very costly, or when you are unsure. Use a canary release for a normal update, and promote step by step, if the numbers stay healthy. Use an A B test when the question is about business impact, not only accuracy. In every case, decide before you start what you will measure, and what result means success, and what triggers a rollback. Releasing without a rollback plan is not a release, it is a gamble.

Summary