AI Hints
Statistics & Math for ML · Lesson 99 of 138

Gradient Descent for Regression

Learning a line.

⏱ 2 min read▶ Video at 1:57:21🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Earlier, we found the best line through the study data with a formula. Now let's find it the way a neural network would. Start with a random line, and improve it, step by step, with gradient descent.

Gradient Descent for Regression
From the video: Gradient Descent for Regression
  • We used a formula before
  • Now learn it step by step
  • Like a neural network

Learning m and b

The loss is the mean squared error. Its gradients for M and B come from the chain rule. Each step, we predict, measure the errors, and nudge M and B downhill. Run it. After two thousand steps, gradient descent reaches the same line as polyfit.

gd_line.pypython
import numpy as np
from study import hours, scores

m, b, lr = 0.0, 0.0, 0.02
for step in range(2000):
    error = (m * hours + b) - scores
    m -= lr * 2 * (error * hours).mean()
    b -= lr * 2 * error.mean()

print(f"gradient descent: m = {m:.2f}, b = {b:.2f}")
print("polyfit:          m = %.2f, b = %.2f" % tuple(np.polyfit(hours, scores, 1)))
Output
gradient descent: m = 5.73, b = 36.86
polyfit:          m = 5.73, b = 36.86
Also uses study.py
import numpy as np

hours = np.array([1, 2, 3, 4, 5, 6, 7, 8])
scores = np.array([42, 50, 53, 61, 64, 72, 74, 85])

Why bother?

For one line, the formula is easier. But gradient descent works for any model, as long as you can compute the gradient of the loss. Neural networks with billions of weights have no neat formula. They're trained with exactly this loop. Predict, measure the error, compute gradients, and step.

keyvalue
Lossmean squared error
Gradientsfrom the chain rule
Looppredict, error, step
Scalesto billions of weights

Summary

Gradient descent learned the same line as the formula, one step at a time. The same loop trains neural networks. Next, speeding it up with small batches.

Key takeaways
  • Predict
  • Measure the error
  • Step downhill
Next up: Stochastic Gradient Descent — small batches