How Kaggle Winners Work

Kaggle is a website where people compete to build the best model, sometimes for big prizes. Winners don't rely on secret algorithms. They follow a very disciplined routine. Learning it will make you better, even if you never enter a contest.

• Competitions for the best model
• No secret algorithms
• A disciplined routine

The winning routine

First, build a validation setup that you can trust, one that mirrors the test data. This matters more than anything else. Second, build a quick baseline, in the first hour. Third, look hard at the data, and the errors, to get ideas for new features. Fourth, run many small experiments, changing one thing at a time, and write down every result. And fifth, near the end, ensemble your best models.

1. Trust validationmirror the test set
2. Fast baselinein the first hour
3. Features from errorsstudy mistakes
4. Ensemblecombine at the end

Lessons worth stealing

Four habits are worth stealing. Trust your cross validation, not the public leaderboard, because many people overfit to it, and then fall down the rankings at the end. Study your worst errors, because they point to your next feature. Log every experiment, since memory is unreliable. And respect the data's structure. If the real test is on future dates, or on new customers, then your validation must be too.

Quick check

Try this one. Your cross validation says point nine zero, but the leaderboard says point eight. Your data is from different hospitals, and the test uses new hospitals. What's the likely problem? A, the model is too simple. B, the same hospitals appear in your train and validation folds. C, you need more trees. D, the leaderboard is broken. The answer is B. That's group leakage. You should split by hospital, using group K fold.

Summary