Repeatable Cleaning

Next month, a new customer file will arrive, with the same mess. If you cleaned this one by hand, you'll have to repeat all of it. A professional writes the cleaning once, as code, and adds checks that shout when something goes wrong.

• Next month, the same mess
• Write it once
• Add checks that shout

A cleaning function

We wrap the steps in one function, chained together, so each new file gets the same treatment. Then come assert statements. These are small tests. No duplicates, sensible ages, tidy cities. If any fail, the program stops with an error, instead of quietly giving you wrong numbers. Run it. All checks pass. That's the difference between an analysis and a reliable pipeline.

import pandas as pd

def clean_customers(df):
    return (df.drop_duplicates()
              .assign(city=lambda d: d["city"].str.strip().str.title(),
                      age=lambda d: d["age"].where(d["age"].between(16, 90)))
              .assign(age=lambda d: d["age"].fillna(d["age"].median())))

df = clean_customers(pd.read_csv("customers.csv"))

assert df.duplicated().sum() == 0
assert df["age"].between(16, 90).all()
assert df["city"].str.istitle().all()
print("All checks passed on", len(df), "customers")

All checks passed on 6000 customers

Missing data, honestly

Think about why data is missing. If ages are missing at random, then filling in the median is fine. But if older customers refused to answer, your averages will be biased, whatever you do. A careful analyst keeps a note of what was filled in, and reports it. Never hide a data problem. Your credibility depends on it.

Summary