Tokenisation

How many words are in, can't? One, or two? What about New York? Or a price like twelve dollars fifty? Splitting text into pieces seems like the simplest thing in the world, until you try to do it properly.

• How many words in can't?
• New York? $12.50?
• Harder than it looks

Why split is not enough

Python's split simply cuts at spaces. Look at the output. Dr dot keeps its dot, and order hash numbers stay stuck to their punctuation. SpaCy's tokeniser knows English. It keeps Dr dot together as one unit, splits can't into ca and n't, and separates the dollar sign. Run it, and compare the two. Each language has its own rules. Chinese has no spaces at all, for example.

import spacy

nlp = spacy.load("en_core_web_sm")
text = "Dr. Lee can't find order #48213! It cost $12.50, didn't it?"

print("Plain split:", text.split()[:7])
print("spaCy:      ", [t.text for t in nlp(text)][:12])

Plain split: ['Dr.', 'Lee', "can't", 'find', 'order', '#48213!', 'It']
spaCy:       ['Dr.', 'Lee', 'ca', "n't", 'find', 'order', '#', '48213', '!', 'It', 'cost', '$']

Levels of tokens

We can split at many levels. Sentences, for summarising. Words, for classic methods. Characters, for spelling and tiny models. And word pieces, which are what transformers use. Remember that a model only ever sees numbers, one for each token. So how you split decides what the model can see. In our Deep Learning course, you built a word piece tokeniser from scratch.

Summary