Do Two Images Look Alike?

A customer complains that our compression has ruined the pictures in the quality report. The engineer replies that the average pixel error is tiny, so nothing's wrong. They're both looking at the same images. Who's right? Often, the human is right, and the simple number is what's misleading.

• Pictures look ruined
• Pixel error says fine
• Who's right?

Brightness against blur

SSIM compares three things, brightness, contrast, and structure, in small windows across the image. This function calculates local averages and variances, using a Gaussian blur. Then it combines them into one score, where one means identical. Our test image has lots of fine detail. We make one copy that's simply brighter, and another that's slightly blurred. Run it. Look at the pixel error. It says that the brighter copy is nearly three times worse. But every detail in it is still there. Now look at SSIM. The brighter copy scores point nine five, and the blurred one only scores point eight two six. So the two numbers disagree, and SSIM agrees with your eye. Blur destroys structure, and a change of brightness doesn't.

import cv2
import numpy as np

def ssim(a, b):
    a, b = a.astype(np.float64), b.astype(np.float64)
    blur = lambda z: cv2.GaussianBlur(z, (11, 11), 1.5)
    ma, mb = blur(a), blur(b)
    va, vb = blur(a * a) - ma * ma, blur(b * b) - mb * mb
    cov = blur(a * b) - ma * mb
    c1, c2 = 6.5025, 58.5225
    top = (2 * ma * mb + c1) * (2 * cov + c2)
    return (top / ((ma * ma + mb * mb + c1) * (va + vb + c2))).mean()

rng = np.random.default_rng(0)
img = cv2.GaussianBlur(rng.integers(0, 255, (200, 200), dtype=np.uint8), (5, 5), 0)
img = np.clip((img * 1.0 - 128) * 3 + 128, 0, 255).astype(np.uint8)

brighter = np.clip(img + 40.0, 0, 255).astype(np.uint8)
blurred = cv2.GaussianBlur(img, (5, 5), 0)
for name, other in [("Brighter", brighter), ("Blurred", blurred)]:
    mse = ((img * 1.0 - other) ** 2).mean()
    print(f"{name:9} MSE {mse:7.1f}   SSIM {ssim(img, other):.3f}")

Brighter  MSE  1530.1   SSIM 0.950
Blurred   MSE   546.8   SSIM 0.826

Which number should you use?

Use the right score for each job. Mean squared error is simple, and it's what many models train on. Peak signal to noise is the same idea on a log scale. It's common in compression papers. SSIM handles structure, and it's cheap. And if you need to match real human opinions, then deep perceptual scores, such as LPIPS, tend to do better, although they're much slower.

Where this is useful

You can use these scores in many places. A test which fails when a camera update changes the images. A check that compression hasn't hurt small details. A way to spot a frozen camera, because every frame scores one. And a quick score for restoration work. All of them run with no labels at all, which makes them very cheap to monitor.

Where SSIM falls short

It's useful, but it has its blind spots. Shift an image by a few pixels, and it can fall sharply, though you'd barely see any change. It works on one channel, so colour problems may pass unnoticed. And it knows nothing about what's important. A tiny change on a bruise may matter far more than a big one in the background. So when a task has a real goal, measure that goal directly, such as the accuracy of the model that reads the image.

Summary