Hard Interview Questions

Here are some questions that have tripped up strong candidates. Try to answer each one before listening. There's more to them than there first seems.

• Strong candidates get tripped up
• Try each one
• More than it seems

Average order value went up. Good news?

Not necessarily. If you lost your small orders, the average rises, although total sales might fall. It's called a mix shift. So ask about total revenue, the number of orders, and the distribution. A good answer always asks what's in the average. Ratios can improve for bad reasons.

Our new feature users spend more. Did it work?

Be careful. People who choose a feature are different to those who don't. They're probably your keenest customers. That's self selection. To answer properly, you'd need an experiment. If that's too late, try matching similar users, or compare users before and after. Never trust a comparison where people picked their own group.

Sign-ups jumped 50 percent overnight. What do you do?

Don't celebrate yet. First, check for a tracking change, or a release. Then check for bots, or for a campaign. Slice by country, device, and source. Check downstream too. Did those users do anything? A real jump shows up everywhere. A fake one shows up in only one number.

How do you pick a sample size?

Work it through with four numbers. The current rate. The smallest effect that's worth finding. The false alarm rate that you'll accept, usually five percent. And the power that you want, usually eighty percent. Then calculate, and divide by daily traffic, to get the length. If it's too long, test a bigger change, or use a less noisy metric.

Summary