Unicode Traps

Scribe's fraud filter blocks any message with the word Paypal. One morning, a scam gets through, and the message plainly says Paypal. You look at it. It's spelled correctly. The filter's working. So what happened? The answer is that text is much stranger than it looks.

• The filter blocks Paypal
• A scam gets through
• It's spelled correctly

Text that lies

Here are two copies of the word cafe. The second uses a plain e, followed by a separate accent character. Normalising puts both into the same form. Next, two copies of Paypal, and we ask for the name of one letter. Then an FI ligature, a single character that some PDFs produce. And the German word for street, and a woman at a laptop. Run it. Both cafes look the same, yet they aren't equal, with lengths of four and five. After normalising, they match. The Paypal comparison is false, and here's why. That letter a is really Cyrillic. That's how the scam got through. The ligature becomes plain letters with NFKC. Lowercase can't match the German spellings, but case fold can. And one small emoji is really three characters long.

import unicodedata as u

a, b = "café", "cafe\u0301"
print(a, b, a == b, len(a), len(b))
print("After NFC:", u.normalize("NFC", a) == u.normalize("NFC", b))

print("Paypal" == "Pаypal")
print(u.name("Pаypal"[1]))

print("finance".lower(), u.normalize("NFKC", "finance"))
print("STRASSE".lower() == "straße".lower())
print("STRASSE".casefold() == "straße".casefold())

flag = "👩‍💻"
print(len(flag), [hex(ord(c)) for c in flag])

café café False 4 5
After NFC: True
False
CYRILLIC SMALL LETTER A
finance finance
False
True
3 ['0x1f469', '0x200d', '0x1f4bb']

A safe cleaning order

Do things in a fixed order. First, decode your bytes properly, as UTF eight. Then normalise, with NFC for most text, or NFKC if you're matching and searching. Case fold, instead of lowercasing. Remove invisible characters. And for security filters, map lookalike letters to a safe form. Log what you change, as these changes can affect your token counts too.

Where people slip up

Some of the most common bugs come from counting. A database limit might use bytes, while Python counts code points, and your screen shows something else. So a message that fits on screen can overflow in storage. Slicing a string can also cut an emoji in half. Decide what you're counting, and keep it clear.

Testing your cleaning

Your cleaning code deserves real tests. Make a small file of nasty text. Include accents in both forms, emoji, right to left text, invisible spaces, and the kind of broken characters that you get from a bad decode. Run it through each step, and check the result. Keep each new surprise from production in that file, so that it can never come back. Cleaning bugs are silent, and they quietly lower every score further down the line.

Summary