Why Now?
Data, chips, and better ideas.
The first artificial neuron was described in nineteen forty three. Backpropagation became famous in the nineteen eighties. And yet, for decades, neural networks were a backwater, and some researchers even avoided the name. Three things had to arrive at the same moment, before everything changed.
- Invented in the 1940s
- A backwater for decades
- Three things had to arrive
Three forces
First, data. The internet gave us billions of labelled photos, words, and clicks. Deep networks are hungry, and at last there was enough food. Second, compute. Graphics chips, built for video games, turned out to be perfect for the matrix maths of neural networks. And third, better ideas. New activation functions, dropout, and better initialisation made deep networks trainable. Any one of these alone wouldn't have been enough. Together, they lit the fuse.
From neurons to GPT
Here's the journey in five steps. In nineteen fifty eight, the perceptron was born. In nineteen eighty six, backpropagation became widely known. In twenty twelve, AlexNet won ImageNet. In twenty seventeen, the transformer was published, in a paper called, attention is all you need. And in twenty twenty two, ChatGPT reached the world. You'll build the core of each of these in this course.
- 1958Perceptron
- 1986Backpropagation
- 2012AlexNet
- 2017Transformer
- 2022ChatGPT
Summary
Data, compute, and better ideas came together, and the rest is history. So where does deep learning beat everything else, and where should you not use it? Let's find out.
Key takeaways- Data fed the networks
- GPUs gave them speed
- Better ideas made depth trainable