Detecting Language
In thirty lines.
Bright Books starts selling in Europe, and messages begin arriving in French, Spanish, and German. Each one needs to go to the right team. How can you recognise a language without knowing any of its words? Look at its letter patterns.
- Selling across Europe
- Messages in many languages
- Letter patterns give it away
Character fingerprints
Each language has its own favourite sequences of letters. English loves THE. German loves SCH. We take a sample of each one, and count its three letter sequences. That's a fingerprint. To detect a language, we count which fingerprint overlaps most with the new text. Run it. All four are right, even though we've only used one sentence of training data for each language, and we've never seen those sentences before. Real detectors use the same idea, with more data.
from collections import Counter
SAMPLES = {
"English": "the book was delivered to the house and they were happy with the service",
"French": "le livre a ete livre a la maison et ils etaient contents du service",
"Spanish": "el libro fue entregado en la casa y ellos estaban contentos con el servicio",
"German": "das buch wurde nach hause geliefert und sie waren mit dem service zufrieden",
}
def grams(text):
t = f" {text} "
return Counter(t[i:i + 3] for i in range(len(t) - 2))
models = {k: grams(v) for k, v in SAMPLES.items()}
def detect(text):
g = grams(text)
return max(models, key=lambda k: sum((g & models[k]).values()))
for text in ["where is my order please", "je voudrais un livre", "donde esta mi libro", "ich will das buch"]:
print(f"{text:<28} {detect(text)}")where is my order please English je voudrais un livre French donde esta mi libro Spanish ich will das buch German
Summary
Letter patterns can identify a language, with no dictionary. Text is messy, and now you can handle it. Next, let's build a classifier from nothing but counting.
Key takeaways- Character trigrams are fingerprints
- Compare overlaps
- Works on very little data