Gradient Descent

Imagine standing on a mountain in thick fog, trying to reach the valley. You cannot see the valley. But you can feel the slope under your feet. So you take a small step downhill. Then you feel the ground again, and take another step. Eventually, you reach the bottom. That is gradient descent, and it trains nearly every model in modern A I.

• Lost on a mountain in fog
• Feel the slope, step downhill
• Trains nearly all modern AI

The valley of loss

Picture every possible line as a point on a landscape. The height is the loss. High ground means a bad line, and the lowest point is the best line. Our model starts somewhere random. The gradient tells us which way is uphill, so we step in the opposite direction. Near the top, the slope is steep, so the steps are big. Near the bottom, it flattens, and the steps shrink. And at the bottom, we stop.

Gradient descent in code

Here it is, in plain code. We start with a slope and start of zero. The learning rate controls the size of each step. In the loop, we compute the error. Then these two lines are the heart of it. They are the gradient, and each one nudges a parameter downhill. Run it, and watch. At step zero, the error is large. By step one hundred, it has fallen a long way. And by five thousand, it has settled at the best line. That is learning, in ten lines, with no libraries.

import numpy as np

rooms = np.array([3, 4, 5, 6, 7], dtype=float)
price = np.array([1.5, 1.9, 2.4, 2.8, 3.3])

slope, start = 0.0, 0.0
learning_rate = 0.01

for step in range(5001):
    error = slope * rooms + start - price
    slope -= learning_rate * 2 * (error * rooms).mean()
    start -= learning_rate * 2 * error.mean()
    if step in (0, 10, 100, 1000, 5000):
        print(f"step {step:5d} slope {slope:.3f} start {start:.3f} "
              f"error {(error ** 2).mean():.5f}")

step     0 slope 0.256 start 0.048 error 6.07000
step    10 slope 0.458 start 0.086 error 0.00075
step   100 slope 0.457 start 0.091 error 0.00071
step  1000 slope 0.452 start 0.119 error 0.00061
step  5000 slope 0.450 start 0.130 error 0.00060

The learning rate

The learning rate is the size of your steps, and it matters enormously. Here we try four values. Watch the output. Too small, at point zero zero zero one, and after two hundred steps we still have not arrived. At point zero zero one, the result is much better. At point zero two, it is better still. But at point zero five, the error explodes. The steps are so big that we leap across the valley, and climb higher each time. Finding this value is one of the great practical skills in machine learning.

import numpy as np

rooms = np.array([3, 4, 5, 6, 7], dtype=float)
price = np.array([1.5, 1.9, 2.4, 2.8, 3.3])

for lr in [0.0001, 0.001, 0.02, 0.05]:
    slope, start = 0.0, 0.0
    for step in range(200):
        error = slope * rooms + start - price
        slope -= lr * 2 * (error * rooms).mean()
        start -= lr * 2 * error.mean()
    final = ((slope * rooms + start - price) ** 2).mean()
    print(f"learning rate {lr:<6} final error {final:.4g}")

learning rate 0.0001 final error 0.6465
learning rate 0.001  final error 0.000741
learning rate 0.02   final error 0.0006474
learning rate 0.05   final error 1.584e+102

The one-line version

You will not usually write gradient descent by hand for a line. Scikit learn does the work in one call. We make the model, and call fit. Then coef gives the slope, and intercept gives the start. They match our hand-made answer. Notice one thing. The shape of rooms has to be a table, with one column, which is why we used reshape. Scikit learn always expects rows and columns.

from sklearn.linear_model import LinearRegression
import numpy as np

rooms = np.array([3, 4, 5, 6, 7]).reshape(-1, 1)
price = np.array([1.5, 1.9, 2.4, 2.8, 3.3])

model = LinearRegression()
model.fit(rooms, price)

print("Slope:", model.coef_[0].round(3))
print("Start:", model.intercept_.round(3))
print("Price with 8 rooms:", model.predict([[8]]).round(2))

Slope: 0.45
Start: 0.13
Price with 8 rooms: [3.73]

Your turn: study hours

Your turn. Pause the video and try this. Here are study hours and exam scores for five students. Write gradient descent yourself, and find the slope and the start. Then predict the score for six hours of study. Think about the steps. Start at zero, loop, compute the error, and nudge both numbers. Take your time. Here is one solution. Twenty thousand small steps, with the same two update lines. The slope is close to six, so each hour adds about six marks. And six hours predicts around eighty one.

import numpy as np

hours = np.array([1, 2, 3, 4, 5], dtype=float)
score = np.array([52, 58, 63, 71, 74], dtype=float)

slope, start = 0.0, 0.0
for step in range(20000):
    error = slope * hours + start - score
    slope -= 0.01 * 2 * (error * hours).mean()
    start -= 0.01 * 2 * error.mean()

print(f"slope {slope:.2f}, start {start:.2f}")
print(f"6 hours predicts {slope * 6 + start:.1f}")

slope 5.70, start 46.50
6 hours predicts 80.7

Summary