Review: Experiments

Most companies get their experiments wrong in quiet ways. This round tests whether you'd spot those mistakes.

• Quiet mistakes
• Most companies
• Would you spot them?

Recap: experiments

Randomisation cancels hidden differences. Plan your sample size first. Don't peek. Correct when testing many things. Check the split, before you read the result. Analyse the unit that you randomised. And cuped uses past behaviour to cut the noise.

Question 1

Question 1. To detect half the effect, how much more data do you need? A, Twice. B, Four times. C, Eight times. D, The same. Pause and choose. The answer is B. Effect size is squared.

Question 2

Question 2. Your 50/50 test shows 50,210 and 49,120 users. What should you do? A, Celebrate. B, Investigate before trusting the result. C, Add more users. D, Ignore it. Pause and choose. The answer is B. That's sample ratio mismatch.

Question 3

Question 3. You randomise users but analyse page views. What happens? A, Better accuracy. B, Far too many false wins. C, Nothing. D, Fewer false wins. Pause and choose. The answer is B. Ours hit forty eight percent.

Question 4

Question 4. What should an A/A test show? A, No data. B, About five percent false alarms. C, Fifty percent. D, Always significant. Pause and choose. The answer is B. If not, your system has a bug.

Question 5

Question 5. What does a Bayesian test let you say? A, The p-value is low. B, There is a 98 percent chance B is better. C, The sample is random. D, Nothing new. Pause and choose. The answer is B. That's what people want to know.

Question 6

Question 6. When is a bandit better than an A/B test? A, A lasting product decision. B, A short campaign. C, Slow outcomes. D, When you need clear evidence. Pause and choose. The answer is B. Earning matters more than learning there.

Summary