Momentum
A rolling ball.
Plain gradient descent is like a careful walker, stopping at every step. Momentum turns it into a rolling ball. The ball builds up speed going downhill, so it moves faster through long gentle slopes, and can roll over small bumps.

- A careful walker
- Or a rolling ball
- Speed builds up
Velocity
With momentum, we keep a velocity. Each step, the velocity is mostly the old velocity, plus the new gradient. Then we move by the velocity. We compare both methods on a long, gently sloping valley. Run it. Plain descent has only reached about eight point seven after a hundred steps. Momentum is almost exactly at ten.
def slope(x):
return 0.02 * (x - 10)
plain = 0.0
for _ in range(100):
plain -= 1.0 * slope(plain)
x, v = 0.0, 0.0
for _ in range(100):
v = 0.9 * v - 1.0 * slope(x)
x += v
print(f"plain: {plain:.2f} momentum: {x:.2f} target: 10")plain: 8.67 momentum: 9.96 target: 10
Modern optimizers
Almost every modern optimizer uses momentum. Adam, the most popular one, combines momentum with a separate step size for each weight, based on how big its gradients have been. A momentum of zero point nine is a common default. In PyTorch, it's one argument to the optimizer.
| key | value |
|---|---|
| Velocity | v = 0.9 v − lr × gradient |
| Move | x = x + v |
| Adam | momentum + per-weight steps |
| Default | momentum 0.9 |
Summary
Momentum keeps a velocity, so descent speeds up on gentle slopes. Adam builds on it. Next, naive Bayes in real code.
Key takeaways- Keep a velocity
- Faster on gentle slopes
- Adam uses it