Review: Practical Deep Learning

Finally, the craft. Getting models trained, tuned, and small enough to use.

• The craft
• Trained and tuned
• Small enough to use

Recap: the craft

Use the learning rate finder. Search at random, not on a grid. Log the code, the settings, and the seed. Pruning removes weights that do little. Quantisation stores weights in eight bits. Distillation teaches a small model from a large one. And mixed precision halves your memory.

Question 1

Question 1. Why does random search beat a grid? A, Few settings really matter. B, It is older. C, It uses less code. D, Grids are illegal. Pause and choose. The answer is A. Random tries new values of each.

Question 2

Question 2. What does dynamic quantisation usually give? A, Larger files. B, About four times smaller, similar accuracy. C, Lower accuracy always. D, More training. Pause and choose. The answer is B. It's a great deal.

Question 3

Question 3. Pruning 95 percent without retraining damaged our model. What's the fix? A, Give up. B, Prune a little, retrain, repeat. C, Add dropout. D, Use a grid. Pause and choose. The answer is B. Gradually works.

Question 4

Question 4. What do soft teacher outputs carry? A, Noise. B, Knowledge about which classes look alike. C, Only the answer. D, Bigger files. Pause and choose. The answer is B. That nine percent is information.

Question 5

Question 5. When is a gradient check worthwhile? A, Standard layers. B, Custom layers and losses. C, Never. D, Only for GPUs. Pause and choose. The answer is B. Standard layers are well tested.

Question 6

Question 6. What does label smoothing punish? A, Slow training. B, Overconfidence. C, Small batches. D, Long names. Pause and choose. The answer is B. It helps nearly every classifier.

Summary