Feature Scaling

Imagine a conversation where one person is shouting through a megaphone, and everyone else is whispering. That is what income in dollars does to house age in years. Many algorithms cannot tell that the big number is just measured in different units, so they listen to the loudest feature.

• One shouting, the others whispering
• Dollars versus years
• Algorithms listen to the loudest

Same volume for everyone

Income runs into the tens of thousands, age into the tens, and rooms is just a small number. On a scale like this, income dominates everything. Scaling fixes that. The most common method is standardisation. For each feature, subtract the mean, then divide by the spread. Afterwards, every feature is centred on zero, with a similar size. They all speak at the same volume, and the model can hear each one fairly.

Three scalers compared

Scikit learn offers three common scalers. Here is a column of salaries, with one huge outlier at the end. Standard scaler subtracts the mean, and divides by the spread. Min max scaler squeezes everything between zero and one. Robust scaler uses the median and the middle of the data. Run it. Min max lets the outlier crush everyone else close to zero. Robust keeps the normal values nicely spread. So, if you have outliers, robust is a strong choice.

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

X = np.array([[52000.0], [64000.0], [48000.0], [71000.0], [900000.0]])

for name, s in [("Standard", StandardScaler()),
                ("MinMax", MinMaxScaler()),
                ("Robust", RobustScaler())]:
    print(f"{name:<9}", s.fit_transform(X).ravel().round(2))

Standard  [-0.52 -0.48 -0.53 -0.46  2.  ]
MinMax    [0.   0.02 0.   0.03 1.  ]
Robust    [-0.63  0.   -0.84  0.37 44.  ]

Who needs scaling?

Which models need this? Anything based on distance, like K nearest neighbours, or support vector machines, needs it badly. Anything using gradient descent, including neural networks, and regularised models, trains far better with it. But tree models do not care at all, because they only ask, is this value above or below a line? Scaling is cheap, so when you are unsure, just scale. And always do it inside a pipeline.

Summary