More Statistical Tests

Statistics has dozens of named tests, and that can feel overwhelming. In practice, you only need a handful. And they all ask the same question. Could this pattern be down to chance?

• Dozens of named tests
• You only need a few
• All ask the same question

Chi-square for categories

Do loyalty members leave less often? Both of these are categories, so we use a chi square test on a table of counts. Run it. Loyalty members leave less, and the p value is tiny. So this isn't luck. But remember, this says nothing about cause. It could be that loyal people joined the scheme in the first place.

import pandas as pd
from scipy import stats

df = pd.read_csv("customers.csv").drop_duplicates()
table = pd.crosstab(df["loyalty"], df["churned"])
print(table.to_string())

chi2, p, _, _ = stats.chi2_contingency(table)
print(f"p-value: {p:.4f}")
print(table.div(table.sum(1), axis=0).round(3).to_string())

churned     0     1
loyalty
0        2695  1205
1        1676   424
p-value: 0.0000
churned      0      1
loyalty
0        0.691  0.309
1        0.798  0.202

Comparing many groups

To compare more than two groups, use ANOVA. It asks whether any group differs from the others. Sales are skewed, so we also run Kruskal Wallis, which doesn't assume a bell curve. Run it. The p values are so tiny that they print as zero, so both agree that regions really do differ. If you've got skewed data, or outliers, these rank based methods are a safe choice.

import pandas as pd
from scipy import stats

sales = pd.read_csv("sales.csv")
daily = sales.groupby(["date", "region"])["revenue"].sum().reset_index()
groups = [g["revenue"].values for _, g in daily.groupby("region")]

print(daily.groupby("region")["revenue"].mean().round(0).to_string())
print("ANOVA p-value:", f"{stats.f_oneway(*groups).pvalue:.2e}")
print("Kruskal p-value:", f"{stats.kruskal(*groups).pvalue:.2e}")

region
Africa      2229.0
Americas    1562.0
Asia        1929.0
Europe      3107.0
ANOVA p-value: 0.00e+00
Kruskal p-value: 0.00e+00

Which test?

Here's your guide. Two group means, use a t test. Two group rates, use a proportions test. Categories, use chi square. Several groups, use ANOVA. Skewed data, use a rank test, or a permutation test. And when in doubt, the bootstrap or a permutation test works for almost anything.

Summary