Creating New Features

Your boss gives you a tip. Buyers do not really care about total rooms. They care about how crowded the home feels. And they pay more to be near big cities. A good model can't invent these ideas on its own, but you can. This is feature engineering, and it often adds more than any fancy algorithm.

• Buyers care about crowding
• And about distance to cities
• You can teach the model that

Better ingredients

Same simple model, better ingredients. First, the score with the original features. Next, we clip outliers, as we learned, and that gives the biggest jump. Now we add ratios. The share of rooms that are bedrooms, people per room, and the log of population. The score moves up a little. Now let's use real knowledge about California. We add the distance from each district to four big cities. Up again. Each step is small, but they add up. Latitude and longitude alone are hard for a linear model, because a city is a point, not a straight line. We taught it what a city is.

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score, KFold

df = fetch_california_housing(as_frame=True).frame
y = df.pop("MedHouseVal")
folds = KFold(5, shuffle=True, random_state=0)
score = lambda d: cross_val_score(LinearRegression(), d, y, cv=folds).mean()

print(f"Original features:      {score(df):.3f}")

new = df.clip(upper=df.quantile(0.99), axis=1)
print(f"Outliers clipped:       {score(new):.3f}")
new["BedroomShare"] = new["AveBedrms"] / new["AveRooms"]
new["PeoplePerRoom"] = new["AveOccup"] / new["AveRooms"]
new["LogPopulation"] = np.log1p(new["Population"])
print(f"With ratio features:    {score(new):.3f}")

new["SF"] = np.hypot(new["Latitude"] - 37.77, new["Longitude"] + 122.42)
new["LA"] = np.hypot(new["Latitude"] - 34.05, new["Longitude"] + 118.24)
new["SD"] = np.hypot(new["Latitude"] - 32.72, new["Longitude"] + 117.16)
new["SJ"] = np.hypot(new["Latitude"] - 37.34, new["Longitude"] + 121.89)
print(f"With distance to cities:{score(new):.3f}")

Original features:      0.604
Outliers clipped:       0.673
With ratio features:    0.679
With distance to cities:0.687

Taming skewed features

Many real numbers are skewed. Most values are small, with a long tail of huge ones. Income, population, and city sizes all look like this. A log transform pulls that tail in. Here we measure the skew of three columns, before and after. A skew near zero is nicely balanced. Run it, and look how much the log brings each one back into shape.

import numpy as np
from sklearn.datasets import fetch_california_housing
from scipy.stats import skew

df = fetch_california_housing(as_frame=True).frame

for col in ["MedInc", "Population", "AveOccup"]:
    print(f"{col:<11} skew {skew(df[col]):7.2f}"
          f"  after log {skew(np.log1p(df[col])):6.2f}")

MedInc      skew    1.65  after log   0.23
Population  skew    4.94  after log  -1.04
AveOccup    skew   97.63  after log   3.88

Which features matter?

More features are not always better. Useless ones add noise, and slow things down. Mutual information measures how much knowing one feature tells you about the target, and it catches curved relationships too. We score every feature, and rank them. Med inc comes first by a long way, and latitude and longitude follow. This gives you a quick view of where the signal is.

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.feature_selection import SelectKBest, mutual_info_regression
from sklearn.model_selection import train_test_split

X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X_tr, _, y_tr, _ = train_test_split(X, y, train_size=4000, random_state=0)

sel = SelectKBest(mutual_info_regression, k=3).fit(X_tr, y_tr)
ranked = sorted(zip(sel.scores_, X.columns), reverse=True)
for score, name in ranked:
    print(f"{name:<11} {score:.3f}")

Longitude   0.376
MedInc      0.370
Latitude    0.307
AveRooms    0.091
AveOccup    0.062
HouseAge    0.025
Population  0.019
AveBedrms   0.014

Quick check

Time for a question. You want to predict delivery time for a pizza. Which new feature would probably help the most? A, the customer's name. B, the distance between shop and home. C, the order number. D, the colour of the box. The answer is B. A good feature carries real meaning about the thing that you are predicting.

Summary