Tensors

Your first morning at Neuron Labs, your new team lead writes one word on the whiteboard. Tensor. Everything in deep learning, every image, every word, and every weight, is stored as a tensor. Master this one thing, and you've mastered the language that all of A I is written in.

• One word on the whiteboard
• Images, words, weights
• The language of AI

Shapes are everything

A tensor is just a box of numbers, with some number of dimensions. A single number has zero dimensions. A list has one. A table has two. And look at the last one. A batch of thirty two colour images, with three colour channels, each two hundred and twenty four pixels square, has four dimensions. Run it. Reading shapes is the number one skill in this field. If you know the shape at every step, then you understand the model.

import torch

scalar = torch.tensor(3.0)
vector = torch.tensor([1.0, 2.0, 3.0])
matrix = torch.zeros(2, 3)
batch = torch.rand(32, 3, 224, 224)

for name, t in [("scalar", scalar), ("vector", vector),
                ("matrix", matrix), ("batch", batch)]:
    print(f"{name:<7} dims {t.ndim}  shape {list(t.shape)}")

scalar  dims 0  shape []
vector  dims 1  shape [3]
matrix  dims 2  shape [2, 3]
batch   dims 4  shape [32, 3, 224, 224]

Tensor maths

Tensors do maths on all of their numbers at once. A plus B adds matching positions. Star multiplies position by position. The at sign is the matrix product, which is a very different thing, and it's the operation at the core of every neural network. Reshape changes the shape, but not the numbers. And mean with dim zero collapses that dimension, giving one mean for each column. Run it, and compare the star result with the at sign result, because beginners mix these up all the time.

import torch

a = torch.tensor([[1., 2.], [3., 4.]])
b = torch.tensor([[10., 20.], [30., 40.]])

print("Add:\n", a + b)
print("Element by element:\n", a * b)
print("Matrix product:\n", a @ b)
print("Reshaped:", a.reshape(4))
print("Mean of each column:", a.mean(dim=0))

Add:
 tensor([[11., 22.],
        [33., 44.]])
Element by element:
 tensor([[ 10.,  40.],
        [ 90., 160.]])
Matrix product:
 tensor([[ 70., 100.],
        [150., 220.]])
Reshaped: tensor([1., 2., 3., 4.])
Mean of each column: tensor([2., 3.])

Broadcasting and shape errors

Broadcasting lets tensors of different shapes work together, by stretching the smaller one. A single tax rate multiplies all of the prices. A set of three brightness values reshapes to line up with colour channels, and it applies to a whole batch of images. And now, you'll meet the most common error in deep learning, a shape mismatch. A three by four matrix can't multiply itself, because the inner numbers don't match. Run it. Transposing fixes it. Whenever you hit this error, print the shapes. That solves it nearly every time.

import torch

prices = torch.tensor([[100.], [200.], [300.]])
tax = torch.tensor([1.1])
print("One number stretched:", (prices * tax).ravel())

images = torch.rand(32, 3, 28, 28)
brightness = torch.tensor([0.5, 1.0, 1.5]).reshape(1, 3, 1, 1)
print("Result shape:", (images * brightness).shape)

wrong = torch.rand(3, 4)
try:
    wrong @ wrong
except RuntimeError as e:
    print("Error:", str(e).split("(")[0].strip())
print("Fixed:", (wrong @ wrong.T).shape)

One number stretched: tensor([110., 220., 330.])
Result shape: torch.Size([32, 3, 28, 28])
Error: mm: mat1 and mat2 shapes cannot be multiplied
Fixed: torch.Size([3, 3])

Summary