Bias, Variance, and More Data

Collecting more data can cost HomeWorth a fortune. So before spending a penny, you want to know, would it even help? Two ideas answer this. Bias and variance. And a single chart can tell you which problem you have.

• More data costs money
• Will it even help?
• Bias and variance will tell us

Bias and variance

Think of predictions as darts, and the true answer as the bullseye. High bias means your darts land together, but in the wrong place. The model is too simple, and it is consistently wrong. High variance means they scatter everywhere. Train it on slightly different data, and you get a very different answer. That is overfitting. The goal is low bias and low variance, and the two usually trade against each other.

The learning curve

A learning curve shows the score as we give the model more and more data. We draw two lines, the train score and the test score. Run it. Read this chart carefully. The two lines come together quickly, and then go flat, at around point six. The gap is tiny, so variance is low. But both lines are stuck well below one. That is the signature of high bias. So buying more data will not help. Our model is simply too simple.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import learning_curve, KFold

X, y = fetch_california_housing(return_X_y=True)
sizes, tr, te = learning_curve(
    LinearRegression(), X, y, train_sizes=np.linspace(0.02, 1, 8),
    cv=KFold(5, shuffle=True, random_state=0), scoring="r2")

plt.figure(figsize=(9, 4.5))
plt.plot(sizes, tr.mean(axis=1), "o-", label="Train")
plt.plot(sizes, te.mean(axis=1), "o-", label="Test")
plt.xlabel("Training rows")
plt.ylabel("R2")
plt.legend()
plt.tight_layout()
plt.savefig("learning_curve.png", dpi=120)
print("Final gap:", round(tr.mean(axis=1)[-1] - te.mean(axis=1)[-1], 3))

Final gap: 0.003

What to do next

Here is the cheat sheet. If you have high bias, with both scores low, then use a more powerful model, or add better features. More data will not help. If you have high variance, with a big gap between train and test, then get more data, simplify the model, or add regularisation. This chart just saved your company a lot of money.

Summary