Data Versioning

Imagine your model was great last month, and today it is worse. What changed? The code, or the data? Without versions of the data, you can never know. Git is excellent for code, but it is not made for big files. In this lesson, you will meet D V C, which gives data the same powers as code.

• Which data trained this model?
• Git cannot hold big files
• DVC versions data like code

How DVC works

The idea is clever. D V C stores your big file in a separate place, called the remote, which can be a folder, or cloud storage. In your Git repository, it saves only a tiny text file, with a fingerprint of the data. When you commit, Git records which version of the data goes with which version of the code. To go back in time, you check out an old commit, and D V C fetches the matching data. Small files in Git, big files in storage, linked by a fingerprint.

1. dvc addfingerprint the data
2. Git tracks the .dvc filetiny text pointer
3. dvc pushbig file goes to storage
4. Checkout + dvc checkouttime travel for data

Versioning a dataset

Let's use it. We initialise D V C, and tell it to track the data file. It creates a small pointer file, and ignores the real data in Git. We commit the pointer, set up a storage remote, and push the data. Now we generate a different version of the data, with another seed, and track it again, and commit. The Git history shows two data versions. And now the time travel. We check out the old pointer, and ask D V C to restore the data. The file is back to its first version, and its fingerprint proves it.

dvc init -q && git commit -qm 'init dvc'
dvc add -q data/churn.csv
git add data/churn.csv.dvc data/.gitignore && git commit -qm 'data v1'
dvc remote add -d storage ../dvcstore
Setting 'storage' as a default remote.
dvc push -q
python src/make_data.py 2 && dvc add -q data/churn.csv && git commit -qam 'data v2'
wrote data/churn.csv with seed 2
git log --oneline
7973f4f data v2
a8b855d data v1
6204f46 init dvc
git checkout -q HEAD~1 -- data/churn.csv.dvc && dvc checkout -q && head -2 data/churn.csv
tenure,charges,calls,contract,churn
34,51.3,2,0,1

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

import sys
from churn import make_data

seed = int(sys.argv[1])
make_data(2000, seed=seed).to_csv("data/churn.csv", index=False)
print("wrote data/churn.csv with seed", seed)

Why this matters

Data versioning gives you three superpowers. Reproducibility, you can rebuild any past model, exactly. Traceability, every model is linked to the data and code that made it, which auditors love. And collaboration, a teammate can fetch the same data with one command, instead of emailing files. Remember the rule. If it can change the result, it must be versioned.

Summary