Fixing a Biased Survey

In nineteen thirty six, an American magazine polled over two million people, and predicted a landslide defeat for Roosevelt. He won by a landslide. The magazine had drawn its names from phone books and car registrations, in the middle of the Great Depression. Size can't fix a biased sample. And that lesson's just as true for your customer survey.

• 1936: two million people polled
• They predicted the wrong winner
• Size can't fix bias

Weighting the responses

Harbor Coffee's survey went out through the app. So sixty percent of replies came from people under thirty five, who only make up thirty five percent of customers. They also happen to be happier. Run it. The raw result says seventy five percent happy. We know the true age mix of our customers, so we can give each answer a weight, which is the real share divided by the sample share. The weighted answer is sixty seven. That's eight points lower, from one line of arithmetic. Pollsters do this constantly.

import numpy as np
import pandas as pd

share_real = {"18-34": .35, "35-54": .40, "55+": .25}
answers = pd.DataFrame({"age": ["18-34"] * 600 + ["35-54"] * 300 + ["55+"] * 100,
                        "happy": [1] * 510 + [0] * 90 + [1] * 195 + [0] * 105 + [1] * 45 + [0] * 55})

print(f"Raw: {answers['happy'].mean():.1%}")
share_sample = answers["age"].value_counts(normalize=True)
answers["weight"] = answers["age"].map(lambda a: share_real[a] / share_sample[a])
print(f"Weighted: {np.average(answers['happy'], weights=answers['weight']):.1%}")

Raw: 75.0%
Weighted: 67.0%

Better surveys

Weighting fixes known differences, like age. It can't fix unknown ones, such as people who are angry being more likely to reply. So ask neutral questions. Keep it short, because long surveys lose people. Check the response rate. And remember that what people say isn't always what they do. Whenever you can, measure behaviour instead.

Summary