Costs and Thresholds

A missed bruise isn't a wasted apple.

⏱ 2 min read▶ Video at 1:39:15🎓 Free

Watch this lesson in the Computer Vision courseWatch on YouTube ↗Full Computer Vision course

A bruised apple which reaches a supermarket can cost a whole crate rejection. Throwing away a good apple costs only twenty cents. These mistakes aren't equal. Yet by default, a model treats them as if they were, with a threshold of point five.

A bruised apple can reject a crate
A wasted apple costs twenty cents
The mistakes aren't equal

Picking a threshold by cost

The model gives a probability. We choose the cut-off. We price each mistake, at three dollars for a miss, and twenty cents for waste. Then we test four thresholds. Run it. At the default of point five, we miss eighteen bruises, and it costs fifty four dollars. Drop to point one, and we only miss two, for a few wasted apples, and the cost falls to about eight dollars. That's a saving of more than eighty percent, with no change to the model. Go too low, at point zero three, and the waste brings the cost back up. There's a sweet spot, and you find it with money, not accuracy.

threshold.pypython

import torch
from patch import make, net, train

torch.manual_seed(0)
model = train(net(), *make(1500))
X, y = make(1000, seed=9)
p = model(X).softmax(1)[:, 1].detach()

miss_cost, waste_cost = 3.00, 0.20
print("Threshold  missed  wasted  cost")
for t in [0.5, 0.3, 0.1, 0.03]:
    bad = p >= t
    missed = (~bad & (y == 1)).sum().item()
    wasted = (bad & (y == 0)).sum().item()
    print(f"{t:>9} {missed:>7} {wasted:>7} ${missed * miss_cost + wasted * waste_cost:.2f}")

Output

Threshold  missed  wasted  cost
      0.5      18       0 $54.00
      0.3      13       4 $39.80
      0.1       2      11 $8.20
     0.03       1      51 $13.20

Also uses patch.py

import numpy as np
import cv2
import torch

def make(n, seed=0, light=1.0, bruise_rate=0.5):
    """Simulated apple photos. Label 1 means bruised."""
    rng = np.random.default_rng(seed)
    X, y = [], []
    for _ in range(n):
        img = np.full((48, 48, 3), (70, 75, 80), np.uint8)
        c = tuple(int(v) for v in rng.integers(19, 30, 2))
        r = int(rng.integers(15, 20))
        cv2.circle(img, c, r, (35, 40, int(rng.integers(150, 225))), -1, cv2.LINE_AA)
        bad = rng.random() < bruise_rate
        if bad:
            at = (c[0] + int(rng.integers(-8, 9)), c[1] + int(rng.integers(-8, 9)))
            cv2.circle(img, at, int(rng.integers(3, 6)), (25, 45, 70), -1, cv2.LINE_AA)
        img = img * light + rng.normal(0, 8, img.shape)
        X.append(img.clip(0, 255) / 255)
        y.append(int(bad))
    X = torch.tensor(np.array(X), dtype=torch.float32).permute(0, 3, 1, 2)
    return X, torch.tensor(y)

def net():
    return torch.nn.Sequential(
        torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
        torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
        torch.nn.Flatten(), torch.nn.Linear(32 * 12 * 12, 2))

def train(model, X, y, epochs=8):
    opt = torch.optim.Adam(model.parameters(), lr=2e-3)
    for _ in range(epochs):
        for i in torch.randperm(len(X)).split(64):
            loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
            opt.zero_grad(); loss.backward(); opt.step()
    return model

def accuracy(model, X, y):
    return (model(X).argmax(1) == y).float().mean().item()

Summary

Set thresholds using real costs. It's the cheapest improvement you'll ever make. Next, the failure that catches nearly every team.

Key takeaways

Mistakes have different costs
Thresholds are business decisions
Cost beats accuracy

Next up: When the Light Changes — the failure that surprises everyone

❮ A Small Bruise Detector When the Light Changes ❯