Forecasting with LSTMs

Power companies need to know tomorrow's demand. Shops need next week's sales. Time series are everywhere, and deep learning can help. But it's also where people fool themselves the most, because a simple trick can look like intelligence.

• Tomorrow's demand, next week's sales
• Time series are everywhere
• People fool themselves here

An LSTM against a baseline

We've got a noisy wave. Each input is the past thirty values, and the target is the next one. The LSTM reads the window, and we take its last output. We train on the first nine hundred points, and test on the rest. Critically, we also compute what you'd score by simply repeating the last value. Run it. The L S T M's error is point one two seven. The lazy baseline gets point one six four. So the model really has learned something, but it isn't dramatic. A famous trap is a forecast chart that looks perfect, where the model is just copying yesterday, one step late. Always compare with the baseline.

import numpy as np
import torch

torch.manual_seed(0)
t = np.arange(1200)
series = np.sin(t * 2 * np.pi / 50) + 0.1 * np.random.default_rng(0).normal(size=1200)
w = 30
X = torch.tensor(np.lib.stride_tricks.sliding_window_view(series[:-1], w), dtype=torch.float32)[..., None]
y = torch.tensor(series[w:], dtype=torch.float32)
tr = 900

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = torch.nn.LSTM(1, 32, batch_first=True)
        self.out = torch.nn.Linear(32, 1)
    def forward(self, x):
        return self.out(self.lstm(x)[0][:, -1]).squeeze(1)

m = Model()
opt = torch.optim.Adam(m.parameters(), lr=.01)
for epoch in range(6):
    for i in torch.randperm(tr).split(32):
        loss = torch.nn.functional.mse_loss(m(X[i]), y[i])
        opt.zero_grad(); loss.backward(); opt.step()

print("LSTM error:", round(((m(X[tr:]) - y[tr:]) ** 2).mean().sqrt().item(), 3))
print("Repeat last value:", round(((X[tr:, -1, 0] - y[tr:]) ** 2).mean().sqrt().item(), 3))

LSTM error: 0.127
Repeat last value: 0.164

Time series rules

Keep test data strictly in the future. Never shuffle across time when you split. Scale using training data only. Forecast several steps ahead, as that's where the real difficulty is. And know that for many business series, simple methods and boosted trees are hard to beat. Deep models shine when you've got thousands of related series.

Summary