Tokens

Computers do not understand letters or words. They only understand numbers. So before a language model can read anything, the text must be turned into numbers. In this lesson, you will see exactly how, using real examples from a real tokenizer.

• Computers only understand numbers
• Text is cut into small pieces
• Each piece gets a number

Cutting text into tokens

The first step is called tokenization. The text is cut into small pieces, called tokens. A token can be a whole word, a part of a word, or even a single character. Here is a real example, from the tokenizer of GPT two. Common words, like the and cat, are single tokens. But a longer word, may be split into pieces. Each token has a number, called its token I D. The model never sees letters. It only sees these numbers.

Rare words are split into pieces

Look what happens with unusual words. The name of a medicine, or a made up word, gets chopped into several smaller pieces. That is clever. The model has a fixed vocabulary of about fifty thousand tokens, but by combining pieces, it can build any word, in any language, even words it has never seen. Spaces matter too. The dot before a token shows that a space came before it.

Why tokens matter to you

Tokens matter, for three practical reasons. First, cost. Most A I services charge by the token, so shorter prompts cost less. Second, limits. Every model has a maximum number of tokens it can handle at once. Third, odd mistakes. Because the model sees tokens, not letters, tasks like counting the letters in a word can be surprisingly hard. As a rule of thumb, one token is about three quarters of an English word.

Summary