AutoML and Big Data
Automation and scale.
If algorithm selection and tuning follow a recipe, can't a machine do that too? It can. And what if your data doesn't even fit in memory? Let's look at both, because they're common questions in real jobs.
- Can a machine do the recipe?
- Data too big for memory
- Two real questions
What AutoML really does
AutoML tools like AutoGluon, FLAML, and many cloud services run a huge bake-off for you. At their core, they do something that you could write yourself. Here's a tiny version. A function tries a list of models, scores each with cross validation, and returns the ranked results. Real tools add smart search, and stacking. Run it. They can give excellent results quickly. But they can't define the problem, collect good data, catch leakage, or explain the result to your boss. They do the part of the job that was already easiest.
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor, RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = fetch_california_housing(return_X_y=True)
X, y = X[:6000], y[:6000]
def auto_ml(models):
results = []
for name, model in models:
results.append((cross_val_score(model, X, y, cv=3).mean(), name))
return sorted(results, reverse=True)
options = [("Ridge", make_pipeline(StandardScaler(), Ridge())),
("Forest", RandomForestRegressor(40, n_jobs=-1, random_state=0)),
("Boost slow", HistGradientBoostingRegressor(learning_rate=0.03, max_iter=300)),
("Boost fast", HistGradientBoostingRegressor(learning_rate=0.1))]
for score, name in auto_ml(options):
print(f"{score:.3f} {name}")0.118 Boost slow 0.104 Boost fast 0.064 Forest -7.282 Ridge
When data does not fit
What about data that's too large for memory? Many models must see all of it at once. But some can learn in small pieces. S G D regressor uses stochastic gradient descent, updating from one small batch at a time. Partial fit trains on each batch, and then forgets it. Here, we stream ten million rows through a tiny memory footprint. Run it. Ten million rows, in seconds, on a laptop. Note also that a sample of your data is often all that you need. For truly massive jobs, there's Spark, and XGBoost can run across many machines.
import numpy as np
import time
from sklearn.linear_model import SGDRegressor
rng = np.random.default_rng(0)
true_w = rng.normal(size=20)
model = SGDRegressor(random_state=0)
start = time.time()
for batch in range(200):
X = rng.normal(size=(5000, 20))
y = X @ true_w + rng.normal(0, 0.1, 5000)
model.partial_fit(X, y)
print(f"Rows seen: {200 * 5000:,} in {time.time() - start:.1f}s")
print("Weight error:", np.abs(model.coef_ - true_w).max().round(4))Rows seen: 1,000,000 in 0.2s Weight error: 0.0025
Summary
AutoML handles the search, but the thinking is still yours. And when data is huge, stream it in batches, or start with a sample. Next, some hard-earned wisdom. The expensive mistakes that real teams keep making.
Key takeaways- AutoML automates the bake-off
- Humans define the problem
- partial_fit streams big data