Choosing Metrics

In colonial India, the story goes, the government offered a reward for every dead cobra. So people started breeding cobras. When the reward ended, they let them all go, and there were more snakes than before. Whether or not it's quite true, every company has its own cobra story.

• A reward for dead cobras
• People bred cobras
• Every company has a cobra story

When a target backfires

Here's a support team, rewarded for closing tickets quickly. We track that target, along with a second number, the share of problems solved first time. Run it. Fast closes rise from sixty to nearly a hundred. The manager gets a bonus. But solved first time falls from seventy nine percent to fifty eight. People were closing tickets without fixing anything. That's Goodhart's law. When a measure becomes a target, it stops being a good measure.

import numpy as np

rng = np.random.default_rng(0)
weeks = 26
calls_closed_fast = 60 + np.arange(weeks) * 1.5 + rng.normal(0, 2, weeks)
solved_first_time = 80 - np.arange(weeks) * 0.8 + rng.normal(0, 1.5, weeks)

print(f"Target metric, fast closes: {calls_closed_fast[0]:.0f} -> {calls_closed_fast[-1]:.0f}")
print(f"Guardrail, solved first time: {solved_first_time[0]:.0f}% -> {solved_first_time[-1]:.0f}%")

Target metric, fast closes: 60 -> 98
Guardrail, solved first time: 79% -> 58%

A healthy set of metrics

So never use one number alone. Choose a north star metric, which captures the value that customers get. Add input metrics, which your team can move this week. And add guardrail metrics, which must not get worse. For Harbor Coffee, that might be monthly repeat customers, then visits per customer, and then waste and staff turnover as guardrails.

What makes a good metric

A good metric is tied to real value. It's hard to game. It moves soon enough for you to act. It's understood by everyone. And someone owns it. Be wary of vanity metrics, such as total sign ups, as they only ever go up. Prefer rates and active users, which can fall, because a number that can't fall can't warn you.

Summary