The ColumnTransformer

A real table has numbers that need filling and scaling, alongside words that need encoding. You could handle each one separately, and glue the results together by hand. Every beginner does this once, and then spends a day finding the bug. Professionals use a single recipe.

• Numbers and words together
• Glue code hides bugs
• One clean recipe

One recipe for every column

A column transformer splits your table into groups of columns. Number columns go down one path, which fills in the gaps and scales them. Word columns go down another, which encodes them. The results are joined back together, and passed to the model. And the entire thing is a single object, that you can fit, cross validate, and save.

1. Raw tablemixed columns
2. Numbers pathfill, then scale
3. Words pathone-hot encode
4. Modellearns from all of it

The full recipe

Here it is in full. We have a table with blanks, numbers, and a city. The numbers path is a small pipeline, which fills with the median, then scales. The words path encodes, and handle unknown ignore tells it not to crash on a city that it has never seen. Column transformer connects each path to its columns. And a final pipeline adds the model. Now look at the last lines. We predict for a house with a missing room count, in Tokyo, which it has never seen. It still works. This pipeline is ready for the real world.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.DataFrame({
    "rooms": [3, 4, np.nan, 6, 5, 4, 3, 7],
    "income": [2.5, 3.1, 4.2, np.nan, 6.0, 3.3, 2.8, 7.1],
    "city": ["Lima", "Oslo", "Lima", "Rome", "Oslo", "Rome", "Lima", "Oslo"],
    "price": [1.2, 1.9, 2.1, 2.8, 3.0, 2.2, 1.3, 3.9],
})
y = df.pop("price")

numbers = make_pipeline(SimpleImputer(strategy="median"), StandardScaler())
words = OneHotEncoder(handle_unknown="ignore")

prep = ColumnTransformer([
    ("num", numbers, ["rooms", "income"]),
    ("cat", words, ["city"]),
])
model = Pipeline([("prep", prep), ("ridge", Ridge())]).fit(df, y)

new = pd.DataFrame({"rooms": [np.nan], "income": [4.0], "city": ["Tokyo"]})
print("Prediction:", model.predict(new).round(2))

Prediction: [2.15]

Summary