Segmentation

A box around an apple tells you where it is. But how large is the bruise? Is it two percent of the apple or twenty? To answer that, we need to know exactly which pixels are bruise. Segmentation labels every pixel in the image. It's how robots pick fruit, and how doctors measure tumours.

• A box isn't enough
• How large is the bruise?
• Label every pixel

Three kinds of segmentation

There are three kinds. Semantic segmentation labels each pixel with a class, so that all apples are one colour. Instance segmentation separates each object, so that apple one and apple two are different. And panoptic does both, for the whole scene, including background. For counting and measuring individual apples, we need instance segmentation.

Instance segmentation

YOLO can do this too, with the seg version of the model. It gives a mask for each object. Run it. There's the person, painted pixel by pixel, not just a box. The mask shape tells us how many objects there are, along with the mask height and width. Now we can measure real shapes.

import cv2
import numpy as np
import matplotlib.pyplot as plt
from skimage import data
from ultralytics import YOLO

model = YOLO("yolo11n-seg.pt")
image = cv2.cvtColor(data.astronaut(), cv2.COLOR_RGB2BGR)
r = model(image, verbose=False)[0]

print("Objects:", [model.names[int(c)] for c in r.boxes.cls])
print("Mask shape:", r.masks.data.shape)

plt.figure(figsize=(5, 5)); plt.imshow(r.plot()[:, :, ::-1]); plt.axis("off")
plt.tight_layout(); plt.savefig("masks.png", dpi=110)

Objects: ['person']
Mask shape: torch.Size([1, 640, 640])

Scoring a mask

How do we score a mask? Pixel accuracy can be misleading when most pixels are background. So we use I o U, or the very similar Dice score. Our prediction here is a bit offset. Run it. Accuracy looks strong, but I o U tells the truth. Dice is popular in medical imaging. Always use one of these for segmentation.

import numpy as np

truth = np.zeros((100, 100), bool)
truth[20:80, 20:80] = True
pred = np.zeros((100, 100), bool)
pred[30:90, 25:85] = True

both = (truth & pred).sum()
iou = both / (truth | pred).sum()
dice = 2 * both / (truth.sum() + pred.sum())
acc = (truth == pred).mean()

print(f"Pixel accuracy {acc:.3f}")
print(f"IoU            {iou:.3f}")
print(f"Dice           {dice:.3f}")

Pixel accuracy 0.830
IoU            0.618
Dice           0.764

Summary