AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Samples and Classes
Statistics & Math for ML · Lesson 125 of 138

Stratified Splits

Keep the class balance.

⏱ 2 min read▶ Video at 2:29:21🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

If only five percent of your data is the rare class, a random train test split might put too few of them in the test set, or too many. Stratified splitting keeps the same class balance in every part.

Stratified Splits
From the video: Stratified Splits
  • 5% rare class
  • Random splits can wobble
  • Keep the balance

stratify=y

We make labels with exactly five percent positives. Then we split three ways. Plain random splits with different seeds, and a stratified split, passing stratify equals Y. Run it. The random test sets get two and a half percent, and seven percent. The stratified one is exactly five percent.

stratify.pypython
import numpy as np
from sklearn.model_selection import train_test_split

y = np.array([1] * 50 + [0] * 950)
X = np.arange(1000).reshape(-1, 1)

for seed in [2, 9]:
    _, _, _, yte = train_test_split(X, y, test_size=0.2, random_state=seed)
    print("random test share:    ", yte.mean())
_, _, _, yte = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
print("stratified test share:", yte.mean())
Output
random test share:     0.025
random test share:     0.07
stratified test share: 0.05

Use it by default

For classification, stratifying is almost always a good idea. Cross validation in scikit learn does it automatically for classifiers, using stratified K fold. For time series, though, never shuffle. Train on the past, and test on the future, or your score will be far too optimistic.

keyvalue
train_test_splitstratify=y
Cross-validationStratifiedKFold
Rare classesmost important
Time seriessplit by time, no shuffle

Summary

Stratified splits keep each class's share the same in training and test data. Use them for classification. Next, what to do when one class is rare.

Key takeaways
  • stratify=y
  • StratifiedKFold
  • Time series: no shuffle
Next up: Imbalanced Classes — when one class is rare