Weight Initialisation

If all your weights start at zero, every neuron will do exactly the same thing, forever. If they start too small, the signal fades away. If they start too big, it explodes. For years, deep networks failed, and part of the reason was simply that they started in the wrong place.

• All zeros: nothing learns
• Too small: the signal fades
• Too large: it explodes

Watching a signal travel

We send a signal through eight layers, and measure how much spread survives at the end. First with tiny weights, then with a smart scheme called Kaiming initialisation, then with large weights. Run it. Too small, and the signal has shrunk to nearly nothing. Too large, and it blows up by a huge factor. The good one holds steady, layer after layer. That's the idea. Each layer should keep the signal at about the same size.

import torch
from torch import nn

torch.manual_seed(0)
x = torch.randn(1000, 100)

for name, scale in [("Too small", 0.01), ("Good", None), ("Too large", 1.0)]:
    out = x
    layers = [nn.Linear(100, 100) for _ in range(8)]
    for layer in layers:
        if scale is None:
            nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")
        else:
            nn.init.normal_(layer.weight, 0, scale)
        out = torch.relu(layer(out))
    print(f"{name:<10} final spread {out.std().item():.3e}")

Too small  final spread 3.056e-02
Good       final spread 7.588e-01
Too large  final spread 4.953e+06

The practical rule

You rarely need to do this yourself, because PyTorch's layers already start with sensible values. The two famous schemes are Xavier, for sigmoid and tanh, and Kaiming, for ReLU. What matters is that you understand the symptom. If deep models aren't learning at all, or the loss turns into not a number, check how signals and gradients are flowing through the layers.

Summary