Finding Duplicate Images
Perceptual hashing.
The farm has two hundred thousand training photos, and many are near copies. Same apple, resized, or with different brightness. Near duplicates waste labelling money, and they can leak between training and test sets. A normal file hash won't help, because changing one pixel changes it completely. We need a hash that sees what a person sees.
- Two hundred thousand photos
- Many near copies
- A hash that sees like a person
Perceptual hash
We shrink the image to thirty two by thirty two, which removes the fine detail. Then a discrete cosine transform keeps the low frequencies, the overall structure. Each of those sixty four numbers becomes a single bit, depending on whether it's above the median. Now we compare a photo, an edited copy with a new size and brightness, and a totally different photo. Run it. The edited copy differs by only four bits, out of sixty four. The different photo differs by thirty four, which is about half, as you'd expect from chance. Compare the bits, and you can search millions of images in a moment.
import cv2
import numpy as np
from belt import belt
def phash(img):
small = cv2.resize(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), (32, 32)).astype(np.float32)
low = cv2.dct(small)[:8, :8]
return (low > np.median(low)).ravel()
a, _, _ = belt(seed=1)
copy = cv2.convertScaleAbs(cv2.resize(a, (320, 240)), alpha=1.2, beta=10)
other, _, _ = belt(seed=7)
print("Bits different, edited copy:", (phash(a) != phash(copy)).sum(), "of 64")
print("Bits different, new photo: ", (phash(a) != phash(other)).sum(), "of 64")Bits different, edited copy: 4 of 64 Bits different, new photo: 34 of 64
Also uses belt.py
import numpy as np
import cv2
def belt(n=7, seed=1, size=(480, 640), bruises=2, light=1.0):
"""Draw a simulated conveyor belt of apples. Returns the image and the true boxes."""
rng = np.random.default_rng(seed)
h, w = size
img = np.full((h, w, 3), (70, 75, 80), np.uint8)
img = cv2.add(img, rng.normal(0, 6, img.shape).clip(0, 30).astype(np.uint8))
boxes, spots, taken = [], [], []
while len(boxes) < n:
r = int(rng.integers(32, 48))
x, y = int(rng.integers(r + 5, w - r - 5)), int(rng.integers(r + 5, h - r - 5))
if any((x - a) ** 2 + (y - b) ** 2 < (r + c + 8) ** 2 for a, b, c in taken):
continue
taken.append((x, y, r))
red = int(rng.integers(150, 225))
cv2.circle(img, (x, y), r, (35, 40, red), -1, cv2.LINE_AA)
cv2.circle(img, (x - r // 3, y - r // 3), r // 4, (90, 90, min(255, red + 30)), -1, cv2.LINE_AA)
boxes.append((x - r, y - r, x + r, y + r))
for x, y, r in taken[:bruises]:
cv2.ellipse(img, (x + r // 3, y + r // 4), (r // 3, r // 4), 20, 0, 360, (25, 45, 70), -1, cv2.LINE_AA)
spots.append((x, y, r))
img = cv2.GaussianBlur(img, (3, 3), 0)
return np.clip(img * light, 0, 255).astype(np.uint8), boxes, spotsSummary
A small fingerprint can catch edited copies. It's also how platforms find banned images. Next, a project that goes wrong in an instructive way.
Key takeaways- Low frequencies capture structure
- Compare the bits
- Remove duplicates before splitting