Synthetic Control

Oslo gets a new menu, and it's the only city that does. There's no control group, and no city that's a good match. Economists studying the Basque Country faced this same problem, and they invented a brilliant solution. If no twin exists, then build one.

• Only Oslo gets the new menu
• No good comparison city
• If no twin exists, build one

A weighted mix of other cities

We've got four other cities. We look only at the weeks before the change, and find the mix of those cities that matches Oslo best. The non negative weights make it a real blend of cities. Then we follow that blend into the period after the change. It's the Oslo that would've existed. Run it. The method picks a half and half blend of two cities. Before the change, the gap is zero, as it should be. After, the gap is about fifteen and a half, and the true effect that we built in is fifteen. We've measured a change in a single city, with no experiment at all.

import numpy as np
from scipy.optimize import nnls

rng = np.random.default_rng(0)
weeks = 60
trend = np.cumsum(rng.normal(.5, 1, weeks))
others = np.column_stack([trend * a + rng.normal(0, 1.2, weeks) + b
                          for a, b in [(1, 50), (.6, 80), (1.4, 30), (.8, 65)]])
oslo = .5 * others[:, 0] + .5 * others[:, 2] + rng.normal(0, .8, weeks)
oslo[40:] += 15

w = nnls(others[:40], oslo[:40])[0]
synthetic = others @ w
print("Weights:", w.round(2).tolist())
print(f"Before the change, gap: {(oslo - synthetic)[:40].mean():+.2f}")
print(f"After the change, gap:  {(oslo - synthetic)[40:].mean():+.2f}  (true 15)")

Weights: [0.49, 0.0, 0.49, 0.01]
Before the change, gap: -0.00
After the change, gap:  +15.55  (true 15)

Using it well

The better the match before the change, the more you can trust it. Test it with a placebo. Pretend the change happened earlier, or in another city, and check that you find nothing. Make sure that the other cities weren't affected. And use many weeks of history. Companies use this for ad campaigns and for city launches.

Summary