The ImageNet Moment

In two thousand and six, a young professor called Fei Fei Li had an unfashionable idea. Everyone was working on better algorithms. She thought that the real problem was data. Her team spent years building ImageNet, over fourteen million labelled images, using workers all over the world. Many colleagues thought that it was a waste of time.

• 2006: an unfashionable idea
• The real problem was data
• Fourteen million labelled images

From a contest to a revolution

From twenty ten, ImageNet ran a yearly contest. Classic methods had an error rate of about twenty eight percent, and were improving slowly. Then in twenty twelve, a network called AlexNet, from Geoffrey Hinton's lab, cut the error to about sixteen percent, trained on two gaming graphics cards. Nothing like that jump had ever been seen. In twenty fifteen, ResNet went below human level on that test. And in just a few years, the whole field had changed.

• 2009ImageNet published
• 2010contest begins, 28% error
• 2012AlexNet, 16%
• 2015ResNet, under 4%
• 2021Transformers arrive

Three things came together

Why then? Three things had finally arrived together. Huge labelled datasets. Graphics cards built for video games, which turned out to be ideal for this maths. And some better training tricks, such as dropout and the ReLU activation. The ideas themselves were decades old. Yann LeCun was reading handwritten cheques with convolutional networks in the nineties. They were just waiting for data and compute.

What this teaches you

There's a lesson in this for your own work. Data was the breakthrough. Sometimes, the unglamorous work matters the most. And an idea that fails today may simply be waiting for its moment. Next time that someone tells you your model needs to be cleverer, ask whether what it really needs is better data.

Summary