Features and Pipelines

A model never sees raw data. It sees features, numbers prepared from the data. Scaling, encoding, combining. And here lies one of the most common bugs in machine learning. If the preparation in production differs, even slightly, from the preparation in training, the model gives wrong answers. This is called training serving skew. Let's learn how to prevent it.

• Models see features, not raw data
• Preparation must match exactly
• Training-serving skew is a classic bug

Where skew comes from

Here is how skew happens. A data scientist prepares features in a notebook, with one piece of code. Later, an engineer rewrites the same steps, for the live service, in another language. Small differences creep in. A different rounding, a different handling of missing values, an old version of a category list. The model receives inputs it never saw in training, and its accuracy falls, with no error message. The cure is simple. Use one single piece of code, for both.

1. Notebookfeatures built once
2. Rewritten for servingsmall differences creep in
3. Different inputsthe model is confused
4. One shared pipelinethe cure

One pipeline for everything

Here is the cure, in code. A pipeline bundles preparation and model into a single object. The column transformer prepares the inputs. Numbers are scaled to a common range, and the contract type is one hot encoded. The pipeline chains it with the model, and we train both together. Then we save the whole pipeline to one file. Later, anywhere, we load it, and give it a raw customer, with no manual preparation. The answer is a churn probability. The preparation travels with the model, so training and serving can never disagree.

import joblib, pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from churn import make_data

df = make_data(2000, seed=1)
X, y = df.drop(columns="churn"), df["churn"]

prep = ColumnTransformer([
    ("numbers", StandardScaler(), ["tenure", "charges", "calls"]),
    ("contract", OneHotEncoder(), ["contract"]),
])
pipe = Pipeline([("prep", prep), ("model", LogisticRegression(max_iter=500))])
pipe.fit(X, y)
joblib.dump(pipe, "model.joblib")

loaded = joblib.load("model.joblib")
new_customer = pd.DataFrame([{"tenure": 3, "charges": 95.0, "calls": 4, "contract": 0}])
print("churn probability:", round(loaded.predict_proba(new_customer)[0][1], 2))

churn probability: 0.82

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Feature stores

Large teams go one step further, with a feature store. It is a shared catalogue of features, computed once, and used everywhere. It has two sides. An offline store holds historical values, for training. An online store serves the latest values in milliseconds, for live predictions. Both are fed by the same definitions, which prevents skew, and saves each team from rebuilding the same features. A popular open source example is Feast. You do not need one on day one, but you should know why they exist.

Summary