Reproducibility

A result that nobody can reproduce is just a rumour. In science, and in engineering, reproducibility is the gold standard. For machine learning, it is surprisingly hard, because of hidden randomness, changing libraries, and changing data. In this lesson, you will learn to close each gap.

• A result you cannot repeat is a rumour
• Hidden randomness
• Four things must be fixed

Four sources of difference

Four things can make two runs differ. First, randomness. Splits, initial weights, and shuffling all use random numbers. Second, data. If the file changed, the model will change. Third, code. A small edit to preparation steps changes everything after it. And fourth, environment. A new library version can change the maths, in small ways. To reproduce a run, you must pin all four, seeds, data version, code version, and environment.

The cheapest fix: seeds

Here is a small experiment about luck. We train the same forest three times, with no seed. Each time, the score is different, because the data split, and the trees, are random. Now we fix the seeds. The score is identical, every time. Seeds are the cheapest reproducibility tool there is, and you should set them in every experiment.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from churn import make_data

df = make_data(1500, seed=1)
X, y = df.drop(columns="churn"), df["churn"]

def score(split_seed, model_seed):
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=split_seed)
    model = RandomForestClassifier(n_estimators=30, max_depth=6, random_state=model_seed)
    model.fit(Xtr, ytr)
    return round(accuracy_score(yte, model.predict(Xte)), 3)

print("no seed:", [score(None, None) for _ in range(3)])
print("seeded :", [score(42, 42) for _ in range(3)])

no seed: [0.871, 0.86, 0.867]
seeded : [0.873, 0.873, 0.873]

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Pinning the environment

The environment needs the same care. Pip freeze lists every installed library, with its exact version. We save the important ones into a requirements file. A teammate, or a server, installs from that file, and gets the same versions. For even stronger guarantees, you package the whole environment into a Docker image, which we will do soon. Here is the head of our own list.

pip freeze | grep -iE '^(scikit-learn|pandas|numpy|mlflow|dvc)=='
dvc==3.67.1
mlflow==3.17.2
numpy==2.5.4
pandas==3.1.0
scikit-learn==1.9.1
pip freeze | grep -iE '^(scikit-learn|pandas|numpy|mlflow|dvc)==' > requirements.txt && wc -l requirements.txt
       5 requirements.txt

A reproducibility checklist

Put it together as a checklist. Set seeds. Version the data. Commit the code. Pin the libraries. Record all four in the experiment tracker, with every run. And, ideally, run training in a container, so the environment is frozen. If a colleague can clone the repository, run one command, and get the same model, you have achieved reproducibility. That is the standard to aim for.

1. Seedsset in every run
2. Dataversioned
3. Code and librariescommitted and pinned
4. Run in a containerfrozen environment

Summary