Inside a Neuron

A real brain cell receives signals from thousands of others, adds them up, and if the total is strong enough, it fires. In nineteen forty three, two scientists turned that into maths. It's almost embarrassingly simple. And yet it's the single building block of every A I you've ever used.

• Brain cells add up signals
• Turned into maths in 1943
• The building block of all AI

One artificial neuron

Here's one neuron. It receives three inputs. Each input travels along a connection with its own weight. A large positive weight means, this input really matters. A negative weight means that it counts against. The neuron adds these together. Then the total passes through an activation function, which bends it. The result goes to the next layer. Weights, a sum, and a bend. Training simply means finding good weights.

Why stacking changes everything

One neuron can only draw a straight line. But a layer does something remarkable. It bends and stretches the whole space. Look at these points, a blue circle inside an orange ring. No straight line can split them. But a hidden layer can lift the middle up, like pushing a tent pole. Now a flat cut separates the two. Each layer folds the data into a form that's easier for the next one. That's what deep networks really do.

Proving it with XOR

Let's prove it. X O R is a pattern that a single layer can't learn, since no straight line separates it. We train one linear layer, and then a network with a hidden layer. Run it. The single layer is stuck, and gets it wrong. Add one hidden layer, and it gets all four right. That's the difference that depth makes.

import torch
from torch import nn

X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = torch.tensor([[0.], [1.], [1.], [0.]])

def train(model):
    torch.manual_seed(0)
    opt = torch.optim.Adam(model.parameters(), lr=0.05)
    for _ in range(800):
        loss = nn.functional.binary_cross_entropy_with_logits(model(X), y)
        opt.zero_grad(); loss.backward(); opt.step()
    return (model(X) > 0).float().ravel().tolist()

torch.manual_seed(0)
print("One layer:", train(nn.Linear(2, 1)))
print("Two layers:", train(nn.Sequential(nn.Linear(2, 8), nn.ReLU(), nn.Linear(8, 1))))
print("Target:    ", y.ravel().tolist())

One layer: [1.0, 0.0, 1.0, 0.0]
Two layers: [0.0, 1.0, 1.0, 0.0]
Target:     [0.0, 1.0, 1.0, 0.0]

Summary