AI Hints
Home›Statistics & Math for ML Tutorial›Likelihood and Overfitting
Statistics & Math for ML · Lesson 109 of 138

Maximum Likelihood

The most likely parameters.

⏱ 2 min read▶ Video at 2:09:30🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A coin lands heads seven times in ten flips. What's your best guess for its chance of heads? Most people say zero point seven. Maximum likelihood explains why that's the right answer, and it's how most models choose their parameters.

Maximum Likelihood
From the video: Maximum Likelihood
  • 7 heads in 10 flips
  • Best guess for P(heads)?
  • Maximum likelihood

The likelihood curve

For every possible value of P, we compute the likelihood, the probability of seeing exactly seven heads in ten flips if P were true. Run it. The curve peaks at exactly zero point seven. That's the value of P that makes our data most likely.

mle.pypython
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import binom

p = np.linspace(0, 1, 201)
likelihood = binom.pmf(7, 10, p)

plt.plot(p, likelihood)
plt.axvline(p[likelihood.argmax()], color="#22d3ee", ls="--")
plt.title(f"Best p = {p[likelihood.argmax()]:.2f}"); plt.xlabel("p")
plt.savefig("likelihood.png")

Likelihood in machine learning

Training a model is usually maximum likelihood in disguise. Minimizing cross entropy is the same as maximizing the likelihood of the labels. And minimizing squared error is the same as maximum likelihood, if the errors are normal. In practice, we maximize the log likelihood, because adding logs is easier than multiplying tiny probabilities.

keyvalue
LikelihoodP(data | parameters)
MLEparameters that maximize it
Cross-entropy= negative log likelihood
Logssums instead of products

Summary

Maximum likelihood picks the parameters that make the observed data most probable. Most training is maximum likelihood. Next, bias and variance.

Key takeaways
  • Likelihood = P(data | θ)
  • Pick the peak
  • Loss = −log likelihood
Next up: Bias and Variance — the trade-off