AI Hints
Statistics & Math for ML · Lesson 121 of 138

Log Transforms

Taming skewed data.

⏱ 2 min read▶ Video at 2:24:46🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Incomes, prices, and website visits are often strongly skewed. A few huge values stretch the scale, and squash everything else together. Taking the logarithm is the most common fix in data science.

Log Transforms
From the video: Log Transforms
  • Incomes, prices, visits
  • A few huge values
  • The log fix

Before and after

On the left are skewed incomes. On the right, the same incomes after NP dot log one P, which is the log of one plus each value, so zeros are safe. Run it. The long tail is gone, and the logged data looks close to a bell curve.

log_transform.pypython
import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
income = rng.lognormal(10, 0.8, 5000)

fig, ax = plt.subplots(1, 2, figsize=(9, 4.5))
ax[0].hist(income, bins=60); ax[0].set_title("Income")
ax[1].hist(np.log1p(income), bins=60); ax[1].set_title("log(1 + income)")
plt.tight_layout(); plt.savefig("log.png")

Thinking in ratios

A log turns multiplying into adding. A pay rise from ten thousand to twenty thousand, and from a hundred thousand to two hundred thousand, are the same step on a log scale, because both are doubling. That often matches how the world works. To get back to the original units, use exp, or exp m one.

keyvalue
np.log1plog(1 + x), safe at 0
Undonp.expm1
Effectpulls in long tails
Meaningequal steps are equal ratios

Summary

Log transforms pull in long right tails, and turn ratios into steps. Next, looking at the correlations between many features at once.

Key takeaways
  • np.log1p
  • Skewed to symmetric
  • np.expm1 to undo
Next up: Correlation Matrices — many features at once