Your Project and Tools

Everything in this course is hands on. We will follow one project from start to finish, a model that predicts which customers will leave a service. In this lesson, you will see how the project is organised, and which tools we use for each job.

• One project, start to finish
• A tidy project layout
• One tool for each job

The project layout

A tidy layout is the first M L Ops habit. A data folder holds datasets, which we will version. A source folder holds the code, for data preparation, training, and serving. A models folder holds trained models. A tests folder holds automatic checks. And at the top, files describe the environment, the pipeline, and the container. Anyone who opens this project, can find everything within seconds, and that is exactly the goal.

The tools we use

Here are our tools. Git versions the code. D V C versions data and models, and defines pipelines. M L flow tracks experiments, and stores models in a registry. Optuna tunes settings. Pytest tests our code and our model. Fast A P I serves the model over H T T P. Docker packages everything, and Kubernetes runs it at scale. And Prometheus watches it in production. Each is popular, open source, and replaceable. The ideas matter more than the brands.

Creating the project

Let's create it. We start a Git repository, and make the folders. We add a small script, that generates our customer data, and run it with a seed, so the result is always the same. The script writes a C S V file, in the data folder. We look at the first lines. Each row is a customer, with tenure, monthly charges, support calls, contract type, and the label, churn, one if they left. This is our raw material.

git init -q -b main
mkdir -p data models tests && ls
data
models
src
tests
python src/make_data.py 1
wrote data/churn.csv with seed 1
head -4 data/churn.csv
tenure,charges,calls,contract,churn
34,51.3,2,0,1
37,46.8,4,0,0
54,80.87,1,1,0
wc -l data/churn.csv
    2001 data/churn.csv

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

import sys
from churn import make_data

seed = int(sys.argv[1])
make_data(2000, seed=seed).to_csv("data/churn.csv", index=False)
print("wrote data/churn.csv with seed", seed)

Summary