Deep Learning Myths

Deep learning has grown so quickly that myths spread faster than facts. Let's clear away the most common ones, before you head into the capstone.

• Myths spread faster than facts
• Clear the common ones
• Before the capstone

Myth: networks work like brains

The word neuron makes people think that these are little brains. They were loosely inspired by biology, in the nineteen forties, and then they went their own way. A real neuron is far more complex. Brains don't appear to use backpropagation. Think of a network as a big function, with adjustable numbers, and you'll be much closer to the truth.

Myth: you need huge data

Many people think that you need millions of examples. Our tiny bruise model needed fifteen hundred. Transfer learning can work with a few hundred. Self supervised methods need no labels at all. What you need depends on how hard the problem is, and on how much you can borrow.

Myth: deeper is always better

For years, people assumed that more layers meant better results. Then researchers found that very deep plain networks did worse, even on training data. ResNet fixed that with shortcuts. But today, most improvements come from better data, and a smaller network often wins once speed and cost matter.

Myth: neural networks are black boxes

People say that you can't see inside. But you've done exactly that. Saliency maps, Grad Cam, attention weights, and gradient checks all look inside. They don't explain everything. Yet you can usually find out what a model's looking at, and that often reveals real bugs.

Myth: transformers replaced everything

Transformers took over language, and they've done very well in vision. So people assume that they're the answer to every problem. But you tested tables yourself, and you saw how a simple method can hold its own there. On small data, a convolution's built in assumptions are a real advantage. And for many tasks, a small classic model is cheaper, and good enough. Choose the tool that fits the data, rather than the one in the headlines.

Myth: high accuracy means a good model

A number on a test set can fool you. If ninety five percent of your labels are one class, a useless model scores ninety five. If the same patient appears in both training and test, your score's inflated. And a model can be accurate, while being badly overconfident. So always look at a baseline, at the split, and at the mistakes. A great score is a reason to get suspicious, and to check.

Summary