AI Hints
Statistics & Math for ML · Lesson 100 of 138

Stochastic Gradient Descent

Small batches.

⏱ 2 min read▶ Video at 1:58:32🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

With a million training examples, computing the exact gradient on all of them, for every single step, is very slow. The trick is to estimate the gradient from a small random batch of examples instead. That's stochastic gradient descent.

Stochastic Gradient Descent
From the video: Stochastic Gradient Descent
  • A million examples
  • Every step is slow
  • Use a small random batch

Mini-batches

We create a hundred thousand points on the line Y equals three X plus two, with noise. Each step uses a random batch of just thirty two points. One pass through the data, called an epoch, is over three thousand quick steps. Run it. After one epoch, the weights are very close to three and two.

sgd.pypython
import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(0, 1, 100_000)
y = 3 * x + 2 + rng.normal(0, 0.1, 100_000)

w, b, lr = 0.0, 0.0, 0.1
for idx in rng.permutation(len(x)).reshape(-1, 32):
    err = (w * x[idx] + b) - y[idx]
    w -= lr * 2 * (err * x[idx]).mean()
    b -= lr * 2 * err.mean()
print(f"after one epoch: w = {w:.2f}, b = {b:.2f}")
Output
after one epoch: w = 3.00, b = 2.00

Noisy, but fast

Each batch gives a noisy estimate of the gradient, so the path wobbles. But the steps are so cheap that it gets there much faster overall. The noise can even help, shaking the model out of bad spots. Batch sizes of thirty two to a few hundred are common. Every big neural network is trained this way.

keyvalue
Batcha small random subset
Epochone pass through the data
Noisewobbly but fast
Typical batch32 to 512

Summary

Stochastic gradient descent estimates the gradient from small random batches. It's noisy, fast, and how big models are trained. Next, backpropagation.

Key takeaways
  • Random mini-batches
  • Epoch = one pass
  • Fast and noisy
Next up: Backpropagation Idea — the chain rule in networks