A Neural Network from Scratch

In nineteen sixty nine, a famous book showed that a single artificial neuron cannot learn even a very simple pattern, called X O R. Funding collapsed. It took years for people to find the answer. Today, you'll build that answer yourself, in twenty lines. It's the seed of every A I system you've heard of.

• A simple pattern stopped a field
• One neuron can't learn XOR
• You'll solve it in twenty lines

Layers of neurons

A neuron is something that you already know. It takes a weighted sum of its inputs, and pushes the result through a curve. That's just logistic regression. A neural network connects many of these in layers. The inputs feed a hidden layer, which feeds the output. Each hidden neuron learns a new feature, on its own. That's the big idea. You no longer have to engineer features by hand, because the network builds them for you.

Solving XOR

X O R is true when exactly one of two inputs is true. No straight line can separate that. Here's a network with one hidden layer of eight neurons. In each step, we go forward, computing the hidden values, then the output. Then come the two gradient lines. That's backpropagation, sending the error backward through the layers, to work out how to nudge each weight. Run it. The error falls from a coin flip to almost nothing. And look at the answers. Nearly zero, one, one, zero. You've just solved the problem that once stalled a whole field.

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

rng = np.random.default_rng(0)
w1 = rng.normal(size=(2, 8))
w2 = rng.normal(size=(8, 1))
sig = lambda z: 1 / (1 + np.exp(-z))

for step in range(20001):
    h = sig(X @ w1)
    p = sig(h @ w2)
    grad2 = h.T @ ((p - y) * p * (1 - p))
    grad1 = X.T @ ((((p - y) * p * (1 - p)) @ w2.T) * h * (1 - h))
    w2 -= 1.0 * grad2
    w1 -= 1.0 * grad1
    if step in (0, 2000, 20000):
        print(f"step {step:>5}  error {((p - y) ** 2).mean():.4f}")

print("Answers:", p.ravel().round(2))

step     0  error 0.2959
step  2000  error 0.0015
step 20000  error 0.0001
Answers: [0.01 0.99 0.99 0.01]

In scikit-learn

Scikit learn includes a simple network too. Here, we use two hidden layers on the digits. Run it. We get strong accuracy, with very little code. For tables of data, boosting usually wins. But for images, sound, and language, deep networks are unbeatable. To really build those, you need the tools in our Deep Learning course, which is the natural next step after this one.

from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)

model = make_pipeline(StandardScaler(),
                      MLPClassifier(hidden_layer_sizes=(100, 50),
                                    max_iter=500, random_state=0))

print("Accuracy:", round(cross_val_score(model, X, y, cv=3).mean(), 3))

Accuracy: 0.949

Summary