AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Optimization
Statistics & Math for ML · Lesson 132 of 138

Momentum

A rolling ball.

⏱ 2 min read▶ Video at 2:37:49🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Plain gradient descent is like a careful walker, stopping at every step. Momentum turns it into a rolling ball. The ball builds up speed going downhill, so it moves faster through long gentle slopes, and can roll over small bumps.

Momentum
From the video: Momentum
  • A careful walker
  • Or a rolling ball
  • Speed builds up

Velocity

With momentum, we keep a velocity. Each step, the velocity is mostly the old velocity, plus the new gradient. Then we move by the velocity. We compare both methods on a long, gently sloping valley. Run it. Plain descent has only reached about eight point seven after a hundred steps. Momentum is almost exactly at ten.

momentum.pypython
def slope(x):
    return 0.02 * (x - 10)

plain = 0.0
for _ in range(100):
    plain -= 1.0 * slope(plain)

x, v = 0.0, 0.0
for _ in range(100):
    v = 0.9 * v - 1.0 * slope(x)
    x += v
print(f"plain: {plain:.2f}   momentum: {x:.2f}   target: 10")
Output
plain: 8.67   momentum: 9.96   target: 10

Modern optimizers

Almost every modern optimizer uses momentum. Adam, the most popular one, combines momentum with a separate step size for each weight, based on how big its gradients have been. A momentum of zero point nine is a common default. In PyTorch, it's one argument to the optimizer.

keyvalue
Velocityv = 0.9 v − lr × gradient
Movex = x + v
Adammomentum + per-weight steps
Defaultmomentum 0.9

Summary

Momentum keeps a velocity, so descent speeds up on gentle slopes. Adam builds on it. Next, naive Bayes in real code.

Key takeaways
  • Keep a velocity
  • Faster on gentle slopes
  • Adam uses it
Next up: Naive Bayes in Code — MultinomialNB