Reproducible Analysis

Your manager asks you to update last year's analysis. You open the folder. Final, final two, final really, and use this one. Which do you trust? Nobody remembers. This happens everywhere, and it's why reproducibility matters more than people think.

• Update last year's analysis
• Final, final two, use this one
• Nobody remembers

A project layout

A simple structure fixes most of this. A read me says what the project is, and how to run it. Requirements fix the library versions. Raw data is never edited by hand. Clean data is always created by code, so you can delete it and rebuild it. Notebooks are for exploring. And reusable functions go in a source folder.

harbor-analysis/
  README.md          what this is, and how to run it
  requirements.txt   exact library versions
  data/raw/          never edited by hand
  data/clean/        created by code
  notebooks/         exploring
  src/               reusable functions
  reports/           final output

Notebooks and scripts

Notebooks are great for exploring, as you can see results right next to the code. But they've got a hidden danger. You can run cells out of order, so a notebook may work only because of a cell which you've since deleted. Once something's useful, move it into a script. And before you share a notebook, restart it, and run it from top to bottom.

Habits of trusted analysts

Use git, so that every change is recorded. Set a random seed, so that results repeat. Write down your assumptions, and where the data came from. Ask a colleague to check your work. And ask yourself one question. If I were hit by a bus, could someone rerun this? If the answer's yes, then you've done it right.

Summary