Activation Functions

Here's a strange fact. Without activation functions, a network with a thousand layers would be no smarter than a network with one. All of that depth would be for nothing. And we can prove it, with a few lines of code.

• A thousand layers, no smarter than one?
• Depth needs a bend
• Let's prove it

Depth without a bend

Here's a network of three linear layers, with no activation between them. Now look at this. We multiply the three weight matrices together into one single matrix. And we compare the output of that one matrix with the whole deep network. Run it. They're identical. Stacking straight lines just gives you another straight line. All of the power of depth comes from the bends.

import torch
from torch import nn

torch.manual_seed(0)
x = torch.rand(100, 2)

deep = nn.Sequential(nn.Linear(2, 16), nn.Linear(16, 16), nn.Linear(16, 1))
out = deep(x)

w = deep[2].weight @ deep[1].weight @ deep[0].weight
b = deep[2].weight @ (deep[1].weight @ deep[0].bias + deep[1].bias) + deep[2].bias
flat = x @ w.T + b

print("Max gap from one linear layer:", (out - flat).abs().max().item() < 1e-5)

Max gap from one linear layer: True

The main activations

Sigmoid was the original, squashing values between zero and one. But when values are large, it goes flat, and the slope nearly vanishes. Tanh is similar, and it's centred on zero. Then came the very simple ReLU. If the input is negative, output zero. If it's positive, pass it through. It's fast, and its slope doesn't vanish, and that one change helped to make deep networks trainable. Today, smoother cousins like GELU are found inside transformers.

What should you use?

The advice is simple. Use ReLU in hidden layers, as your default. For transformers, GELU is the common choice. On the output layer, the task decides. Use nothing for regression. Use softmax for one of many classes. And use sigmoid for yes or no, or for several labels at once. In PyTorch, classification losses apply that last step for you, so your model just outputs raw logits.

Summary