Inside Dropout

Dropout randomly switches off half of a layer's outputs. That sounds like it should break things. If half of the signal disappears, wouldn't all of the later layers get a weaker input? So why doesn't everything just shrink? There's a small trick hidden inside.

• Half of the outputs switched off
• Why doesn't everything shrink?
• A small hidden trick

Dropout up close

We pass ten thousand ones through dropout. Run it. About half have been set to zero, as expected. But look at the average. It's still close to one! How? The values that survive get doubled. So half are zero, half are two, and the average stays the same. That's the hidden trick. Later layers see the same overall signal size. And in evaluation mode, nothing's dropped at all, and nothing's doubled. So training and testing line up on average. It's also why you can think of dropout as training thousands of slightly different networks, and then averaging them when you test.

import torch

torch.manual_seed(0)
drop = torch.nn.Dropout(0.5)
x = torch.ones(10000)

print("Training, zeros:", (drop(x) == 0).float().mean().item())
print("Training, mean:", round(drop(x).mean().item(), 2))
drop.eval()
print("Testing, zeros:", (drop(x) == 0).float().mean().item())

Training, zeros: 0.5045999884605408
Training, mean: 0.98
Testing, zeros: 0.0

Using dropout well

How much should you use? Point one or point two in transformers. Up to point five on big fully connected layers. Rarely between convolution layers, as there, neighbouring pixels share information, so dropping single values does little. Dropout can also fight with batch norm, so take care when you use the two together. And if your model is underfitting, take dropout out.

Summary