Missing Values

HomeWorth's new partner sends over a dataset from an estate agent. You open it, and your heart sinks. Blank cells everywhere. Customers skipped questions, sensors failed, and old systems lost things. Welcome to real data. Scikit learn will refuse to run on most models with blanks like these, so you have a decision to make.

• A messy file arrives
• Blank cells everywhere
• Your model will refuse to run

Finding the holes

First, find out how bad it is. Here is a small table with some blanks. Is N A marks every blank, and sum counts them for each column. That gives us a quick report. Two rooms are missing, one income, and one city. Always ask this question first. How much is missing, and in which columns? If a column is ninety percent empty, you may simply drop it. If it is two percent, you can fill it in.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "rooms":  [3, 4, np.nan, 6, 5, np.nan],
    "income": [2.5, 3.1, 4.2, np.nan, 6.0, 3.3],
    "city":   ["Oslo", "Lima", None, "Oslo", "Lima", "Oslo"],
})

print(df.isna().sum().to_string())
print()
print("Share missing:", round(df.isna().mean().mean(), 2))

rooms     2
income    1
city      1

Share missing: 0.22

Filling the holes

Filling in missing values is called imputation. Simple imputer can use the mean, or the median of the column. The median is safer when there are extreme values, because one mansion will not drag it upwards. K N N imputer is smarter. It finds the most similar complete rows, and copies from them. Run it, and compare what each method chose for the same blank cells. There is no universally correct answer, so you test them on your validation score.

import numpy as np
from sklearn.impute import SimpleImputer, KNNImputer

X = np.array([[3, 2.5], [4, 3.1], [np.nan, 4.2],
              [6, np.nan], [5, 6.0], [np.nan, 3.3]])

mean = SimpleImputer(strategy="mean").fit_transform(X)
median = SimpleImputer(strategy="median").fit_transform(X)
knn = KNNImputer(n_neighbors=2).fit_transform(X)

print("Mean:  ", mean[2].round(2), mean[3].round(2))
print("Median:", median[2].round(2), median[3].round(2))
print("KNN:   ", knn[2].round(2), knn[3].round(2))

Mean:   [4.5 4.2] [6.   3.82]
Median: [4.5 4.2] [6.  3.3]
KNN:    [3.5 4.2] [6.   4.55]

Missing can be a signal

Here is an expert trick. Sometimes, the fact that a value is missing tells you something. If a customer refuses to state their income, that may itself predict something. So we keep that information. Setting add indicator to true fills in the blank, and also adds a new column that says, this one was missing. Look at the output. The second column is one wherever the income was blank. Now the model can use the blank as a clue.

import numpy as np
from sklearn.impute import SimpleImputer

income = np.array([[2.5], [np.nan], [4.2], [np.nan], [6.0]])

imputer = SimpleImputer(strategy="median", add_indicator=True)
print(imputer.fit_transform(income))

[[2.5 0. ]
 [4.2 1. ]
 [4.2 0. ]
 [4.2 1. ]
 [6.  0. ]]

Summary