Graphs and Sound

Not everything is an image or a sentence. Molecules, social networks, and road maps are graphs. And speech is a wave. Deep learning handles these too, and you'll be surprised at how much you already know about them.

• Molecules and social networks
• Speech is a wave
• You already know the ideas

Message passing on a graph

A graph is a set of nodes and connections. A graph neural network has one simple idea. Each node updates itself by listening to its neighbours. That's called message passing. Here's a small graph, and node zero starts with a signal. We repeat the mixing step. Run it. The signal spreads along the connections, step by step, and each node gathers information from further away. Real graph networks use learned weights, but this is the heart of it. They're used for drug discovery, fraud detection, and predicting traffic.

import torch

edges = torch.tensor([[0, 1], [1, 2], [2, 3], [3, 0], [0, 2]])
n = 4
A = torch.zeros(n, n)
A[edges[:, 0], edges[:, 1]] = 1
A = A + A.T + torch.eye(n)
A = A / A.sum(1, keepdim=True)

x = torch.tensor([[1.0], [0.0], [0.0], [0.0]])

for step in range(3):
    x = A @ x
    print(f"step {step + 1}:", x.ravel().round(decimals=2).tolist())

step 1: [0.25, 0.33000001311302185, 0.25, 0.33000001311302185]
step 2: [0.28999999165534973, 0.2800000011920929, 0.28999999165534973, 0.2800000011920929]
step 3: [0.2800000011920929, 0.28999999165534973, 0.2800000011920929, 0.28999999165534973]

Turning sound into a picture

Sound is a wave of numbers. A neat trick converts it into a picture, called a spectrogram, showing which frequencies are present at each moment. We make a pure tone of four hundred and forty hertz. S T F T transforms it. Run it. Sixteen thousand samples become a small image, and the loudest frequency is right at four hundred and forty. Now we can apply a normal convolution network. Whisper, a famous speech recognition model, works on exactly this kind of picture.

import torch
from torch import nn

torch.manual_seed(0)
t = torch.linspace(0, 1, 16000)
wave = torch.sin(2 * torch.pi * 440 * t)

spec = torch.stft(wave, n_fft=400, hop_length=160,
                  window=torch.hann_window(400), return_complex=True).abs()

print("Sound samples:", wave.shape[0])
print("Spectrogram:  ", list(spec.shape))
print("Loudest frequency bin:", spec.mean(1).argmax().item(),
      "=", spec.mean(1).argmax().item() * 16000 // 400, "Hz")

conv = nn.Conv2d(1, 8, 3)
print("Now it is an image:", list(conv(spec[None, None]).shape))

Sound samples: 16000
Spectrogram:   [201, 101]
Loudest frequency bin: 11 = 440 Hz
Now it is an image: [1, 8, 199, 99]

Summary