Data Is Everything

There is a famous saying in computing. Garbage in, garbage out. For learning machines, it is even more true. A model is only as good as the data it learns from. In this lesson, you will learn where training data comes from, why it must be clean, and how bias sneaks in.

• Garbage in, garbage out
• Where does training data come from?
• How bias sneaks in

Where training data comes from

So where does the data come from? For language models, from enormous collections of text. Books, articles, websites, encyclopaedias, forums, and computer code. For image models, from huge collections of pictures, each paired with a caption. For speech models, from many hours of recorded voices. The bigger and more varied the collection, the more the model can learn.

Cleaning the data

But raw data is messy. Before training, teams clean it. They remove duplicates, because repeated text makes a model memorise. They filter out spam and low quality pages. They remove private details, and harmful content. Then they split the data into three parts. Most of it is for training. A smaller part is for checking progress, called validation. And a final part is kept locked away, for the final exam, called the test set.

1. Collectgather huge amounts
2. Cleanremove duplicates and junk
3. Filterremove private and harmful content
4. Splittrain, validate, test

Train, validate, test

A typical split looks like this. About eighty percent for training, where the model learns. About ten percent for validation, where we check progress, and tune settings. And about ten percent for testing, a fair final exam, on data the model has never seen. Keeping the test set secret is the only honest way to know how well the model really works.

How bias sneaks in

Now a serious point. A model learns whatever patterns the data contains, including unfair ones. If a hiring dataset mostly contains successful men, the model may learn to prefer men. If photos of doctors are mostly one group, the model may assume only that group are doctors. The machine is not prejudiced. It simply mirrors the data. So careful teams audit their data, and test models for fairness.

Summary