Encoding Categories

Your data has a column called city. Lima, Oslo, Rome. A model can only multiply numbers, so what is Oslo times zero point three? It means nothing. We have to translate the words into numbers, and if we do it carelessly, we quietly teach our model something false.

• What is Oslo times 0.3?
• Models only read numbers
• Careless encoding teaches lies

One-hot encoding

Why not just number them? Lima one, Oslo two, Rome three? Because that says Rome is three times Lima, which is nonsense. Instead, we use one hot encoding. We make one new column for each category. Each row gets a one in its own column, and zeros everywhere else. No false order. Every city stands alone. Here, three colours become three columns.

One-hot in code

In scikit learn, one hot encoder does the work. We fit it on the city column. Sparse output false simply gives us an ordinary table to look at. Feature names out shows the new column names. And there is our table, with exactly one, in each row. Three cities, three columns.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame({"city": ["Lima", "Oslo", "Rome", "Oslo"]})

enc = OneHotEncoder(sparse_output=False)
table = enc.fit_transform(df[["city"]])

print(enc.get_feature_names_out())
print(table)

['city_Lima' 'city_Oslo' 'city_Rome']
[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]
 [0. 1. 0.]]

When order is real

But sometimes, order is real. Small, medium, and large really do have a ranking. Here, we use an ordinal encoder, and we tell it the order ourselves. Be careful. If you leave out the categories, it sorts alphabetically, and you get large, medium, small, which is wrong. Now small is zero, medium is one, and large is two.

import pandas as pd
from sklearn.preprocessing import OrdinalEncoder

df = pd.DataFrame({"size": ["small", "large", "medium", "small"]})

enc = OrdinalEncoder(categories=[["small", "medium", "large"]])
print(enc.fit_transform(df[["size"]]).ravel())

[0. 2. 1. 0.]

Too many categories

What about a column with five hundred zip codes? One hot would add five hundred columns. A clever alternative is target encoding. It replaces each zip code with the average price seen for that zip code, so five hundred categories become a single, very useful number. This is powerful, but also dangerous, because it uses the target, which is a recipe for leakage. Scikit learn's target encoder protects you, by using cross fitting inside.

import numpy as np
import pandas as pd
from sklearn.preprocessing import TargetEncoder

rng = np.random.default_rng(0)
zip_code = rng.integers(0, 500, 3000).astype(str)
effect = {z: rng.normal(0, 1) for z in set(zip_code)}
price = np.array([effect[z] for z in zip_code]) + rng.normal(0, .5, 3000)

X = pd.DataFrame({"zip": zip_code})
enc = TargetEncoder(target_type="continuous", random_state=0)
out = enc.fit_transform(X, price)

print("Unique zip codes:", X["zip"].nunique())
print("Columns after encoding:", out.shape[1])
print("Example values:", out[:3].ravel().round(2))

Unique zip codes: 499
Columns after encoding: 1
Example values: [-0.71  2.38 -1.5 ]

Summary