Regression for Understanding

Machine learning asks, what will happen? Statistics asks, why? A regression can do both. And when you read it the statistician's way, it tells you how much each factor matters, while holding everything else still.

• ML asks what
• Statistics asks why
• Hold everything else still

Reading a regression table

Stats models gives you a full table. Each coefficient says, for one more unit of this, how much does spending change, if the others stay equal? The standard error shows how sure we are. And the p value tests if it could be zero. Run it. Each extra visit is linked to higher spend. And look at age. It has a coefficient that's near zero, with a large p value. So age doesn't seem to matter here. Read one row at a time.

import pandas as pd
import statsmodels.formula.api as smf

df = pd.read_csv("customers.csv").drop_duplicates().dropna()
df = df[df["age"].between(16, 90)]

model = smf.ols("avg_spend ~ visits_per_month + age + loyalty", data=df).fit()
print(model.summary2().tables[1][["Coef.", "Std.Err.", "P>|t|"]].round(3).to_string())
print("R squared:", round(model.rsquared, 3))

                   Coef.  Std.Err.  P>|t|
Intercept         10.014     0.172  0.000
visits_per_month   0.249     0.016  0.000
age               -0.002     0.004  0.683
loyalty           -0.009     0.098  0.927
R squared: 0.041

Controlling for a confounder

Here's why controls matter. We build a simulation where the true effect of a loyalty card is zero. But keen customers both get the card, and spend more. So a naive comparison will mislead. Run it. The naive model says that the card adds a lot. After adding keenness, the effect falls to around zero, which is the truth. Regression only fixes confounders that you measured, though, so hidden ones remain a risk.

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(0)
n = 5000
keen = rng.normal(0, 1, n)
card = (keen + rng.normal(0, 1, n) > 0).astype(int)
spend = 10 + 3 * keen + 0 * card + rng.normal(0, 1, n)
df = pd.DataFrame({"spend": spend, "card": card, "keen": keen})

print("Naive:   ", round(smf.ols("spend ~ card", df).fit().params["card"], 2))
print("Adjusted:", round(smf.ols("spend ~ card + keen", df).fit().params["card"], 2))

Naive:    3.36
Adjusted: 0.0

Logistic regression for yes or no

For yes or no outcomes, like churn, we use logistic regression. Coefficients become odds ratios. Below one lowers the odds, and above one raises them. Run it. Each extra visit lowers the odds of leaving, and so does loyalty. Those are clear, explainable numbers for the founder.

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

df = pd.read_csv("customers.csv").drop_duplicates().dropna()
df = df[df["age"].between(16, 90)]

m = smf.logit("churned ~ visits_per_month + loyalty + age", data=df).fit(disp=0)
odds = np.exp(m.params).round(3)
print(pd.DataFrame({"odds ratio": odds, "p": m.pvalues.round(4)}).to_string())

                  odds ratio       p
Intercept              1.032  0.7894
visits_per_month       0.795  0.0000
loyalty                0.568  0.0000
age                    1.002  0.5157

Summary