Testing Many Things at Once

A proud analyst says the new app design lifted three metrics out of forty. It looks like a great result. But think about what a five percent false alarm rate really means. If you test forty things, how many alarms will go off, even if nothing happened?

• Three metrics went up
• Out of forty
• How many by luck?

Forty metrics, no real effect

We simulate two groups, with forty metrics, and no real effect at all. Every metric is pure noise. We test each one, and count the results under point zero five. Then the Bonferroni rule, which simply divides that threshold by the number of tests. Next, we add a real effect to four of the metrics, and we compare three methods. Run it. With no real effect, three metrics still look significant. That's the proud analyst's result, from pure noise. Bonferroni correctly finds none. With four real effects, the plain test finds seven, so three of those are false. Bonferroni and Benjamini Hochberg both find exactly four.

import numpy as np
from scipy import stats

rng = np.random.default_rng(1)
n_metrics, n_users = 40, 2000
a = rng.normal(0, 1, (n_users, n_metrics))
b = rng.normal(0, 1, (n_users, n_metrics))
p = stats.ttest_ind(a, b).pvalue
print("Metrics tested:", n_metrics, " True effects: 0")
print("Significant:", (p < 0.05).sum())
print("With Bonferroni:", (p < 0.05 / n_metrics).sum())

b[:, :4] += 0.15
p = stats.ttest_ind(a, b).pvalue
print("Now 4 real effects")
print("Plain test finds:", (p < 0.05).sum())
print("Bonferroni finds:", (p < 0.05 / n_metrics).sum())
print("Benjamini-Hochberg finds:", stats.false_discovery_control(p).__lt__(0.05).sum())

Metrics tested: 40  True effects: 0
Significant: 3
With Bonferroni: 0
Now 4 real effects
Plain test finds: 7
Bonferroni finds: 4
Benjamini-Hochberg finds: 4

Two ways to correct

These two corrections answer different questions. Bonferroni protects you against even one false alarm. It's simple and strict, and with lots of tests, it can miss real effects. In this run, it didn't, but the effects were clear. Benjamini Hochberg controls what share of your discoveries are false. It's less strict, so it's the usual choice when you're exploring many metrics.

The garden of forking paths

You don't need to run forty formal tests to get into trouble. Every time that you try a different filter, a different date range, or a different segment, you're running another test. Nobody sees all those paths, and that may include you. So write down your main metric before you begin. Treat everything else as a clue, and not as a result. Then confirm any surprising clue with a fresh test.

A sensible way to report

When you've got many metrics, a simple structure helps. One primary metric decides the launch. A few guardrail metrics check that nothing's been damaged. Secondary metrics help you to explain what happened, and you label them as exploratory. Then report every metric that you tested, including the boring ones. Showing only the winners is how false results get into company decisions.

Summary