Review: How Networks Learn

Before the capstone, let's make this knowledge stick. Pause on each question, and commit to an answer. Getting one wrong now is a gift, because you'll remember the correction.

• Make it stick
• Pause and commit
• Wrong answers are a gift

Recap: learning

A network is layers of simple maths. Autograd finds the slope of the loss for each weight. The optimiser steps downhill. The learning rate sets how far. Activations bend the function. Normalisation keeps signals steady. And regularisation stops memorising.

Question 1

Question 1. What should the starting loss be with ten classes? A, Zero. B, About 2.3. C, About 10. D, One. Pause and choose. The answer is B. That's random guessing.

Question 2

Question 2. What's the fastest check for a training bug? A, Train for a week. B, Overfit one small batch. C, Buy a GPU. D, Read a paper. Pause and choose. The answer is B. It should reach almost zero.

Question 3

Question 3. Why does a model behave differently after model.eval()? A, New weights. B, Batch norm and dropout change behaviour. C, It loads faster. D, It trains more. Pause and choose. The answer is B. Forgetting it is a classic bug.

Question 4

Question 4. Why do surviving values get scaled up in dropout? A, To keep the average signal the same. B, To add noise. C, To save memory. D, To go faster. Pause and choose. The answer is A. Half zero, half doubled.

Question 5

Question 5. Why sample learning rates on a log scale? A, Tradition. B, 0.001 and 0.01 differ as much as 1 and 10. C, It is faster. D, It uses less memory. Pause and choose. The answer is B. That's how networks respond.

Question 6

Question 6. What does a sudden spike in gradient norm warn you about? A, Nothing. B, Training is about to go wrong. C, Faster training. D, Better accuracy. Pause and choose. The answer is B. Log it.

Summary