From ELIZA to Today
Sixty years in ten minutes.
In nineteen sixty six, Joseph Weizenbaum built a program called ELIZA at MIT, which pretended to be a therapist, by turning your words back into questions. It had no understanding at all. Yet his own secretary asked him to leave the room, so that she could talk to it in private. He was shocked. We've been falling for talking machines for sixty years.
- 1966: ELIZA
- No understanding at all
- People confided in it anyway
The road to today
The nineteen sixties were about rules. In the nineties, statistics took over, with IBM's translation work, and the famous joke that each time you fire a linguist, performance goes up. In twenty thirteen, Word two Vec gave us meaning as numbers. In twenty seventeen, Attention Is All You Need introduced the transformer. In twenty eighteen, BERT changed search. And in twenty twenty two, ChatGPT put it all in front of the public.
- 1966ELIZA, rules
- 1990sstatistics
- 2013Word2Vec
- 2017the Transformer
- 2022ChatGPT
What really changed
Look at the pattern. Each step gave up on a human writing the rules, and let data do more. First rules, then counts, then embeddings, and then attention. Researcher Rich Sutton called this the bitter lesson. General methods, with more data and compute, beat clever hand built knowledge, again and again. And yet, as you've seen in this course, the old tools still have their place.
Summary
Sixty years, from rules to learning. So what can these models still not do?
Key takeaways- We've always been fooled by talk
- Each era needed fewer rules
- Old tools still have a place