AI Hints
Home›Statistics & Math for ML Tutorial›Intervals and the Bootstrap
Statistics & Math for ML · Lesson 54 of 138

Bootstrap Anything

Median interval.

⏱ 2 min read▶ Video at 1:02:28🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

The best thing about the bootstrap is that it works for almost any statistic. The median, a percentile, a correlation, or even a model's accuracy. Just swap in a different function.

Bootstrap Anything
From the video: Bootstrap Anything
  • Any statistic
  • Median, percentile, accuracy
  • Swap the function

Seeing the bootstrap distribution

Here are house prices, which are skewed, so we want the median. We bootstrap the median ten thousand times, and draw a histogram of the results, with lines at the interval's ends. Run it. The histogram shows the range of plausible medians.

boot_median.pypython
import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(1)
prices = rng.lognormal(12.5, 0.5, 200) / 1000

meds = [np.median(rng.choice(prices, 200)) for _ in range(10_000)]
low, high = np.percentile(meds, [2.5, 97.5])

plt.hist(meds, bins=50)
plt.axvline(low, color="#22d3ee"); plt.axvline(high, color="#22d3ee")
plt.title(f"Median price: {low:.0f}k to {high:.0f}k")
plt.savefig("boot_median.png")

Bootstrap a model

In machine learning, you can bootstrap your test set to get an interval for a model's accuracy. If one model scores eighty six percent and another scores eighty seven, but the intervals overlap a lot, the difference might just be luck. The bootstrap keeps you honest.

keyvalue
Any statisticjust change the function
Skewed datano normal assumption
Model accuracyresample the test set
Overlapmaybe just luck

Summary

The bootstrap gives intervals for medians, percentiles, and model scores, with no formula. Next, a new topic, hypothesis testing.

Key takeaways
  • Swap the statistic
  • Plot the results
  • Works for model scores
Next up: Hypothesis Testing — innocent until proven