AI Hints
Statistics & Math for ML · Lesson 98 of 138

The Learning Rate

Step size.

⏱ 2 min read▶ Video at 1:56:16🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

The learning rate controls how big each step is. It's the most important setting in training a model. Too small, and learning takes forever. Too big, and you overshoot the valley, and bounce right out of it.

The Learning Rate
From the video: The Learning Rate
  • How big is each step?
  • Too small: forever
  • Too big: overshoot

Three learning rates

We run gradient descent on the same valley, with three learning rates, and plot X after each step. Run it. Zero point zero two crawls slowly. Zero point three gets to three quickly. And one point zero five overshoots further each time, and flies away.

learning_rate.pypython
import matplotlib.pyplot as plt

def path(lr, steps=25):
    x, xs = 0.0, [0.0]
    for _ in range(steps):
        x = x - lr * 2 * (x - 3)
        xs.append(x)
    return xs

for lr in [0.02, 0.3, 1.05]:
    plt.plot(path(lr), "o-", ms=4, label=f"lr = {lr}")
plt.axhline(3, color="#ffffff", ls="--"); plt.ylim(-4, 10); plt.legend()
plt.title("x after each step"); plt.savefig("lr.png")

Choosing it in practice

Common starting points are zero point zero one, or zero point zero zero one. If the loss explodes, lower it. If the loss falls very slowly, raise it. Many modern optimizers, like Adam, adapt the step size for each weight automatically, and many training runs slowly shrink the learning rate over time.

keyvalue
Too smallslow progress
Too bigbounces or explodes
Start0.01 or 0.001
Adamadapts per weight

Summary

The learning rate sets the step size. Too small is slow, and too big diverges. Next, using gradient descent to learn a line.

Key takeaways
  • Small: slow
  • Big: diverges
  • Watch the loss
Next up: Gradient Descent for Regression — learning a line