Data Pipelines

Behind every good model is a data pipeline, that nobody notices, until it breaks. Data arrives messy, late, duplicated, and in strange formats. A pipeline turns it into clean, trustworthy tables. Since models are only as good as their data, operating data pipelines well is half of M L Ops. Let's see the pattern, and a real clean up.

• Data arrives messy
• Pipelines make it trustworthy
• Half of MLOps is data

ETL versus ELT

Two patterns are common. In E T L, you extract data from sources, transform it, on a separate engine, and load only the clean result into the warehouse. In E L T, you extract, and load the raw data first, into a powerful warehouse, and transform it there, with S Q L. E L T has become popular, because storage is cheap, raw data is kept for later reuse, and transformations are easy to rerun, and to version. Either way, the principles are the same. Land the raw data unchanged, clean it in clear steps, and test the output.

Cleaning messy data

Here is a small, realistic clean up. The raw file has typical problems. A duplicated row. Plan names with different capital letters, and stray spaces. A missing monthly charge. A date that is not a date. And a negative charge, which cannot be right. The code removes the duplicate, normalises the plan names, turns bad dates into missing values, treats negative charges as missing, and fills missing charges with the median of the same plan. Finally, it prints a small report, rows out, bad dates, and missing charges. Reports like this one, run on every load, are the beginning of data quality monitoring.

import io
import pandas as pd

raw = io.StringIO("""customer_id,plan,monthly_charge,signup
101,Basic,29.9,2025-03-01
102,basic ,,2025-04-11
102,basic ,,2025-04-11
103,PREMIUM,89.0,not a date
104,Premium,-5.0,2025-06-30
105,Basic,35.5,2025-07-02
""")

df = pd.read_csv(raw)
print("rows in :", len(df))

df = df.drop_duplicates()
df["plan"] = df["plan"].str.strip().str.lower()
df["signup"] = pd.to_datetime(df["signup"], errors="coerce")
df["monthly_charge"] = df["monthly_charge"].where(df["monthly_charge"] > 0)
df["monthly_charge"] = df["monthly_charge"].fillna(df.groupby("plan")["monthly_charge"].transform("median"))

report = {"rows out": len(df), "bad dates": int(df["signup"].isna().sum()), "still missing charge": int(df["monthly_charge"].isna().sum())}
print(report)
print(df.to_string(index=False))

rows in : 6
{'rows out': 5, 'bad dates': 1, 'still missing charge': 0}
 customer_id    plan  monthly_charge     signup
         101   basic            29.9 2025-03-01
         102   basic            32.7 2025-04-11
         103 premium            89.0        NaT
         104 premium            89.0 2025-06-30
         105   basic            35.5 2025-07-02

Habits of reliable pipelines

Reliable pipelines share some habits. They are idempotent, running twice gives the same result, so a retry is always safe. They are incremental, processing only new data, when possible. They validate data at the boundaries, with contracts, and fail loudly. They keep raw data, and every transformation step, so that results can be rebuilt. They record lineage, which table came from which. And they alert on freshness, when expected data does not arrive on time, because silence is the hardest failure to notice.

Summary