Backtesting Forecasts

Your model did brilliantly on the last eight weeks. But those weeks were unusually calm. How would it have done over Christmas? One test period is a single roll of the dice. A serious forecaster tests on many.

• Brilliant on the last eight weeks
• Those weeks were calm
• One test is one roll of the dice

Rolling backtest

We step through time. At each cut, we train only on the past, and forecast the next twenty eight days, then move forward a month, and do it again, six times. We score a simple baseline, and the smoothing model, every time. And we record the worst result too, not just the average. Run it. Smoothing has about half the error on average. But look at the worst month. Its worst error is close to the baseline's worst. That's a much more honest story than one lucky score. Before you promise anything to a manager, you want to know how bad a bad month gets.

import numpy as np
import pandas as pd
from statsmodels.tsa.holtwinters import ExponentialSmoothing

sales = pd.read_csv("sales.csv", parse_dates=["date"])
y = sales.groupby("date")["revenue"].sum().asfreq("D")

errors = {"same day last week": [], "smoothing": []}
for cut in range(len(y) - 28 * 6, len(y), 28):
    train, test = y[:cut], y[cut:cut + 28]
    naive = np.tile(train.iloc[-7:].values, 4)[:len(test)]
    model = ExponentialSmoothing(train, trend="add", seasonal="mul", seasonal_periods=7).fit()
    errors["same day last week"].append((abs(test.values - naive) / test.values).mean())
    errors["smoothing"].append((abs(test.values - model.forecast(len(test)).values) / test.values).mean())

for name, e in errors.items():
    print(f"{name:<20} mean {np.mean(e):.1%}, worst {np.max(e):.1%}")

same day last week   mean 4.7%, worst 5.9%
smoothing            mean 2.4%, worst 5.3%

Backtesting rules

Follow these rules. Never let future data leak into training, including through scaling, or through features. Match the test length to the real decision, so that a monthly order uses a month. Include seasons that matter, such as holidays. Compare everything with a simple baseline. And check error by group, as a good total can hide terrible forecasts for small cafes.

Summary