Learning While Earning

You're testing three versions of an offer. A normal A/B test gives each of them equal traffic until the end, even once one looks like a clear loser. It feels wasteful. Picture a gambler facing three slot machines, wondering how long to keep trying each one. That's why this is called a bandit problem.

• Three versions of an offer
• Equal traffic feels wasteful
• A gambler at three machines

Thompson sampling

The method is beautifully simple. For every visitor, we draw a possible rate for each offer, from our current beliefs. We show the offer with the highest draw. Then we update what we believe. Weak offers get tried less and less. Run it. Look at how the traffic went. The best offer got almost all of it, and the worst was barely shown. And it earned around five hundred more conversions than an equal split would've done. It's learning and earning at the same time.

import numpy as np

rng = np.random.default_rng(0)
true = [0.10, 0.12, 0.15]
wins, losses = np.ones(3), np.ones(3)
shown = np.zeros(3, int)

for visitor in range(20000):
    arm = np.argmax(rng.beta(wins, losses))
    reward = rng.random() < true[arm]
    wins[arm] += reward
    losses[arm] += 1 - reward
    shown[arm] += 1

print("Shown:", shown.tolist())
print(f"Conversions: {int(wins.sum() - 3):,}, an equal split gets about {20000 * np.mean(true):,.0f}")

Shown: [99, 1326, 18575]
Conversions: 2,940, an equal split gets about 2,467

A/B test or bandit?

So do bandits replace A/B tests? No. Use an A/B test when you need a clear, lasting answer, or when the effect takes a while to appear. Use a bandit for short lived decisions, such as a headline, or a seasonal offer, where earning now matters more than learning for later. Bandits are also harder to analyse afterwards.

Summary