Forecasting with Features

A surprisingly strong way to forecast is to forget about special time series models, and turn the problem into the kind of table you already know. Our powerful boosting model can then do the rest. All it takes is a clever use of the past.

• Turn time into a table
• Boosting does the rest
• Use the past as features

Lags beat a baseline

Shift creates lag features. Lag one is yesterday's value, lag seven is the same day last week, and mean seven is the average of the past week. Be careful. We shift before the rolling mean, so that today's value never leaks in. Then we split by time, with the last sixty days as the test set. And we always compare against simple baselines. Run it. Tomorrow equals today, is our first baseline. Same day last week, is the second. Our model beats them both. If it could not, it would be useless.

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(1)
days = pd.date_range("2024-01-01", periods=730, freq="D")
y = (np.linspace(100, 160, 730) + 12 * np.sin(2 * np.pi * days.dayofweek / 7)
     + 20 * np.sin(2 * np.pi * days.dayofyear / 365) + rng.normal(0, 4, 730))
df = pd.DataFrame({"y": y}, index=days)

df["lag_1"] = df["y"].shift(1)
df["lag_7"] = df["y"].shift(7)
df["mean_7"] = df["y"].shift(1).rolling(7).mean()
df["weekday"] = df.index.dayofweek
df = df.dropna()

train, test = df[:-60], df[-60:]
cols = ["lag_1", "lag_7", "mean_7", "weekday"]
model = HistGradientBoostingRegressor(random_state=0).fit(train[cols], train["y"])

print("Tomorrow equals today:", round(mean_absolute_error(test["y"], test["lag_1"]), 2))
print("Same day last week:   ", round(mean_absolute_error(test["y"], test["lag_7"]), 2))
print("Our model:            ", round(mean_absolute_error(test["y"], model.predict(test[cols])), 2))

Tomorrow equals today: 7.71
Same day last week:    5.74
Our model:             5.03

Testing in time order

Ordinary cross validation would let us train on the future, which is cheating. Time series split keeps things in order. Each fold trains on the past, and tests on what comes after. Look at the output. The training window grows, and the test is always later. This is often called walk forward validation, and it's the only honest way to test a forecast.

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

t = np.arange(12)

for i, (train, test) in enumerate(TimeSeriesSplit(n_splits=4, test_size=2).split(t), 1):
    print(f"Fold {i}: train {train.min()}-{train.max()}  test {test.tolist()}")

Fold 1: train 0-3  test [4, 5]
Fold 2: train 0-5  test [6, 7]
Fold 3: train 0-7  test [8, 9]
Fold 4: train 0-9  test [10, 11]

Classic forecasting tools

There are also specialised tools. A R I M A models a series based on its own past values. Prophet, from Meta, handles trends and holidays easily. And neural networks can learn across thousands of series at once. In many real competitions, boosted trees with good lag features perform at the very top, so that's a great place to start.

Summary