Review: Architectures

Each great architecture solved one clear problem. If you remember the problem, you'll remember the design.

• Each solved one problem
• Remember the problem
• Remember the design

Recap: architectures

Convolutions share weights over space. ResNet adds shortcuts, so that very deep networks can train. Recurrent nets carry memory, and L S T M protects it with gates. Attention lets each token look at all of the others. Transformers stack attention with positions. Autoencoders compress. Diffusion models learn to remove noise.

Question 1

Question 1. What problem did ResNet solve? A, Slow disks. B, Very deep networks being hard to train. C, Too much colour. D, Small images. Pause and choose. The answer is B. Shortcut connections fixed it.

Question 2

Question 2. Why do RNN gradients vanish? A, One step is repeated many times. B, Too much data. C, Wrong labels. D, Small files. Pause and choose. The answer is A. LSTM gates help.

Question 3

Question 3. Why do transformers need positional information? A, Attention has no sense of order. B, To save memory. C, To add colour. D, For speed. Pause and choose. The answer is A. Otherwise word order is lost.

Question 4

Question 4. On tabular data, what usually wins? A, Transformers. B, Gradient boosted trees. C, GANs. D, Diffusion. Pause and choose. The answer is B. Ours scored 0.84 against 0.76.

Question 5

Question 5. An LSTM forecast scored 0.127. What must you compare with? A, Nothing. B, Repeating the last value. C, A random number. D, Yesterday's news. Pause and choose. The answer is B. Ours scored 0.164.

Question 6

Question 6. How do recommenders predict a rating? A, Dot product of user and item vectors. B, Sorting by name. C, Word counts. D, Random choice. Pause and choose. The answer is A. People and items become vectors.

Summary