Data Validation

Garbage in, garbage out. A model trained on broken data will be broken too, and often silently. A negative price. A missing column. A category nobody has seen before. The cure is to check the data automatically, every time, before it reaches the model. Let's write such checks.

• Broken data, broken model
• Check automatically, every time
• Fail early, with clear messages

A data contract in code

Here is a data contract, written with a library called pandera. For each column, we state the type, and the rules. Tenure must be between zero and one hundred. Charges must be above zero. Calls cannot be negative. The contract type must be zero, one, or two. And the label must be zero or one. First, we validate good data, and it passes. Then we damage the data on purpose, a negative charge, and an invalid contract. And validate again, collecting all the failures. The report names the column, the rule that broke, and the bad value.

import pandas as pd
import pandera.pandas as pa
from churn import make_data

schema = pa.DataFrameSchema({
    "tenure": pa.Column(int, pa.Check.in_range(0, 100)),
    "charges": pa.Column(float, pa.Check.gt(0)),
    "calls": pa.Column(int, pa.Check.ge(0)),
    "contract": pa.Column(int, pa.Check.isin([0, 1, 2])),
    "churn": pa.Column(int, pa.Check.isin([0, 1])),
})

good = make_data(500, seed=1)
schema.validate(good)
print("good data: passed")

bad = good.copy()
bad.loc[0, "charges"] = -5.0
bad.loc[1, "contract"] = 7
try:
    schema.validate(bad, lazy=True)
except pa.errors.SchemaErrors as err:
    print("bad data: failed")
    print(err.failure_cases[["column", "check", "failure_case"]].to_string(index=False))

good data: passed
bad data: failed
  column           check  failure_case
 charges greater_than(0)          -5.0
contract isin([0, 1, 2])           7.0

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

What else to check

Types and ranges are only the start. Check that no important column is missing, or full of blanks. Check that the share of each category is plausible. Check that the number of rows is within the usual range. And check that today's data resembles the training data, which leads to drift detection, a topic for later in the course. Put these checks at the start of every pipeline, so that bad data stops the line, instead of poisoning the model.

Summary