When the World Is Curved

Income and house price do not follow a perfectly straight line. Real life bends. Yet linear regression can only draw straight things. Or can it? There is a beautiful trick that lets a straight line model fit curves, without changing the algorithm at all.

• Real life bends
• Lines only draw lines
• A beautiful trick

Bending a line

Here is the trick. The model only needs to be straight in its weights. So we invent new features. Alongside x, we add x squared, x cubed, and so on. The model still multiplies each one by a weight, and adds them together. But in the original picture, the result is a curve. Watch what happens as we add more powers. A low degree is too stiff. A medium one fits nicely. And a very high degree goes wild, chasing every single point. Remember this picture, because it is about to become very important.

Polynomial features in code

Scikit learn makes this easy. Polynomial features creates the new columns, and a pipeline joins it to linear regression, so that the two act as a single model. Here is a curved dataset. We try degree one, three, and five. Run it. A straight line explains part of the curve, degree three much more, and degree five nearly all of it. So, should we always use a higher degree? It looks perfect here. But there is a serious trap.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

x = np.linspace(0, 6, 40).reshape(-1, 1)
y = np.sin(x).ravel() + 0.5 * x.ravel()

for degree in [1, 3, 5]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(x, y)
    print(f"degree {degree}: R2 = {model.score(x, y):.3f}")

degree 1: R2 = 0.353
degree 3: R2 = 0.986
degree 5: R2 = 1.000

Summary