Tables: A Reality Check
Where boosting still wins.
Deep learning conquered images, speech, and language. So surely it's the best for spreadsheets too? Most business data lives in tables. Let's run a fair test, and see. The answer surprises a lot of deep learning fans.
- It conquered images and speech
- Surely tables too?
- A fair test
Network against boosting
Here's the California housing dataset. First, a gradient boosting model, with no tuning at all. One line. Then a neural network, with scaling, and twenty five epochs. Run it. Boosting scores about point eight four. The network gets about point seven six. More code, more tuning, and a lower score. Research confirms this. On small and medium tables, tree methods usually beat neural networks. Trees handle odd scales, and sharp thresholds, naturally. So choose the right tool, rather than your favourite tool.
import torch
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = fetch_california_housing(return_X_y=True)
Xa, Xb, ya, yb = train_test_split(X, y, random_state=0)
print("Boosting R2:", round(r2_score(yb, HistGradientBoostingRegressor(random_state=0).fit(Xa, ya).predict(Xb)), 3))
sc = StandardScaler().fit(Xa)
A, B = [torch.tensor(sc.transform(v), dtype=torch.float32) for v in (Xa, Xb)]
Y = torch.tensor(ya, dtype=torch.float32)
torch.manual_seed(0)
net = torch.nn.Sequential(torch.nn.Linear(8, 64), torch.nn.ReLU(), torch.nn.Linear(64, 64), torch.nn.ReLU(), torch.nn.Linear(64, 1))
opt = torch.optim.Adam(net.parameters(), lr=1e-3)
for _ in range(25):
for i in torch.randperm(len(A)).split(256):
loss = torch.nn.functional.mse_loss(net(A[i]).squeeze(1), Y[i])
opt.zero_grad(); loss.backward(); opt.step()
print("Network R2:", round(r2_score(yb, net(B).detach().squeeze(1)), 3))Boosting R2: 0.835 Network R2: 0.759
When networks win on tables
There are real exceptions. Networks do better when tables also contain text or images. They shine with huge data, with millions of rows, such as at big tech companies. They're useful for learning embeddings of categories with many values, such as a product I D. And they work online, updating with each new example.
| Situation | Prefer |
|---|---|
| Small or medium table | Boosted trees |
| Text or images included | Network |
| Hundreds of millions of rows | Network |
| Many-valued categories | Embeddings |
Summary
Use boosting on tables first. But there's one kind of table where embeddings have changed everything. Recommendations.
Key takeaways- Boosting usually wins on tables
- Trees need less tuning
- Mix of data favours networks