Case Study: Ship the Redesign?

Remember our A/B test of the new app design? B won, with a significant p value, and the product team's already celebrating. They've booked a launch date. The founder asks you one question. Will this actually make us more money? And when you look deeper, things become less clear.

• B won the A/B test
• Launch date booked
• Will it make more money?

Step one: more than one number

We've only looked at conversions so far. So let's look at spending as well. Run it. Conversion rises from point one zero seven to point one two two. That's the win we saw. Spend per user goes from point nine eight to one point zero six. So far, so good. But look at the last line. Spend per buyer has fallen, from nine seventeen to eight seventy six. More people are buying, but each of them is buying less. So the new design pulls in smaller orders. This is why you should never look at just one metric.

import pandas as pd

ab = pd.read_csv("ab_test.csv")
t = ab.groupby("group").agg(users=("user", "size"), buy=("converted", "mean"), spend=("spend", "mean"))
print(t.round(3).to_string())
buyers = ab[ab.converted == 1].groupby("group")["spend"].mean().round(2)
print("Spend per buyer:", buyers.to_dict())

       users    buy  spend
group
A       4005  0.107  0.984
B       3995  0.122  1.065
Spend per buyer: {'A': 9.17, 'B': 8.76}

Step two: is the money real?

The founder's question is about spending per user. Spending is very skewed, with lots of zeros, so we bootstrap. Run it. The gain is eight cents per user. But the range runs from minus six cents, to plus twenty three. It includes zero. So conversion is significant, while revenue isn't. The new design might make us more money, or it might not. We can't tell from this test. The team's celebration was about a different number than the one that pays the bills.

import numpy as np
import pandas as pd

ab = pd.read_csv("ab_test.csv")
a, b = ab[ab.group == "A"].spend.values, ab[ab.group == "B"].spend.values
rng = np.random.default_rng(0)
gaps = [rng.choice(b, len(b)).mean() - rng.choice(a, len(a)).mean() for _ in range(5000)]
low, high = np.percentile(gaps, [2.5, 97.5])
print(f"Extra spend per user: {b.mean() - a.mean():.3f}")
print(f"95% range: {low:.3f} to {high:.3f}")

Extra spend per user: 0.081
95% range: -0.061 to 0.229

Step three: what would it be worth?

Now translate that into a business decision. With two hundred thousand users a month, what's each possible lift worth per year, compared with sixty thousand dollars to build? Run it. Even a tiny two cent lift is worth forty eight thousand a year. At the middle of our range, it's hundreds of thousands. The cost is small, and the risk seems to be small too. So the size of the opportunity changes the answer. Even with uncertainty, this bet looks good.

users_a_month = 200000
for lift in [0.02, 0.15, 0.30]:
    print(f"Lift ${lift:.2f} per user: ${lift * users_a_month * 12:,.0f} a year")
print("Build cost: $60,000")

Lift $0.02 per user: $48,000 a year
Lift $0.15 per user: $360,000 a year
Lift $0.30 per user: $720,000 a year
Build cost: $60,000

What you recommend

Here's the page. Recommendation, ship it, but not as the team expected. Roll out to twenty percent first, and measure revenue per user. Why? Conversion rose significantly, but spend per buyer fell, and the revenue gain isn't yet proven. The upside is much greater than the cost. Also investigate why baskets got smaller. Maybe the new design hides add ons. Fixing that could turn this into a clear win.

Summary