XGBoost and LightGBM
The competition winners.
In twenty fourteen, a student named Tianqi Chen released a boosting library that was much faster and stronger than anything before it. It began winning competition after competition. It was called X G Boost. Then Microsoft released LightGBM, and Yandex released CatBoost. Together, these three still rule tabular data.
- 2014: XGBoost arrives
- It starts winning everything
- LightGBM and CatBoost follow
The two champions
Both libraries use the same simple commands as scikit learn. We set up X G Boost and LightGBM with similar settings, then train each, and time them. Run it. Both give excellent results, close to the best we have seen, and each trains in seconds. The exact winner can change from dataset to dataset, so professionals usually try both, and move on.
import time
import lightgbm as lgb
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
X, y = fetch_california_housing(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
models = {
"XGBoost": xgb.XGBRegressor(n_estimators=600, learning_rate=0.05,
max_depth=6, subsample=0.8),
"LightGBM": lgb.LGBMRegressor(n_estimators=600, learning_rate=0.05,
num_leaves=63, verbose=-1),
}
for name, m in models.items():
start = time.time()
m.fit(X_tr, y_tr)
mae = mean_absolute_error(y_te, m.predict(X_te)) * 1e5
print(f"{name:<9} MAE ${mae:,.0f} {time.time() - start:.1f}s")XGBoost MAE $28,824 1.2s LightGBM MAE $28,689 9.7s
How to choose
X G Boost is the veteran, widely supported, and very reliable. LightGBM grows trees leaf by leaf, and it is often the fastest, especially on big data. CatBoost shines when you have many categories, and it needs less tuning. And scikit learn's hist gradient boosting is a great starting point, because you already have it. In practice, the gap between them is small. Good features matter much more.
Stacking models
Here's one final trick from the winners. Stacking combines different kinds of model. A forest, a boosted model, and K nearest neighbours each make predictions. Then a simple model learns how much to trust each one. Because the models make different mistakes, the team can beat each individual. Run it. The stack is competitive with the best. In real business projects, the small gain is rarely worth the extra complexity, but in a competition, it can win prizes.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import (RandomForestClassifier, StackingClassifier,
HistGradientBoostingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
base = [
("forest", RandomForestClassifier(50, random_state=0)),
("boost", HistGradientBoostingClassifier(random_state=0)),
("knn", make_pipeline(StandardScaler(), KNeighborsClassifier())),
]
stack = StackingClassifier(base, LogisticRegression(max_iter=2000), cv=3)
for name, m in base + [("STACK", stack)]:
print(f"{name:<7} {cross_val_score(m, X, y, cv=3).mean():.4f}")forest 0.9596 boost 0.9596 knn 0.9631 STACK 0.9736
Summary
You now hold the same tools that win competitions. X G Boost, LightGBM, and stacking. Trees and boosting are done, and you've reached a big milestone. Next, a very different and elegant idea, the street between two classes.
Key takeaways- XGBoost, LightGBM, CatBoost
- The gap between them is small
- Stacking combines different models