Real Tokenisers

Our Tiny GPT read one character at a time. That's simple, but wasteful. Real models read word pieces, called tokens. It's why a chatbot's price is quoted per token, and why they sometimes struggle to count the letters in a word. Today, you'll build the algorithm that does this splitting.

• Tiny GPT read characters
• Real models read word pieces
• That's why tokens cost money

Byte pair encoding

The algorithm is called B P E, for byte pair encoding. Start with every word as separate letters. Find the pair that appears together most often. Merge that pair into a new token. Then repeat. Run it. First, it merges e and r, or l and o, the most common pairs. Then whole pieces such as low and new appear, automatically. Nobody told it about prefixes or suffixes. Common words become single tokens, and rare words split into pieces. This is the same idea used by G P T models.

from collections import Counter

text = "low lower lowest new newer newest"
words = [list(w) + ["_"] for w in text.split()]

def top_pair(words):
    pairs = Counter()
    for w in words:
        pairs.update(zip(w, w[1:]))
    return pairs.most_common(1)[0][0]

def merge(words, a, b):
    out = []
    for w in words:
        new, i = [], 0
        while i < len(w):
            if i < len(w) - 1 and w[i] == a and w[i + 1] == b:
                new.append(a + b); i += 2
            else:
                new.append(w[i]); i += 1
        out.append(new)
    return out

for step in range(6):
    a, b = top_pair(words)
    words = merge(words, a, b)
    print(f"merge {a!r} + {b!r}")
print(words[2])

merge 'w' + 'e'
merge 'l' + 'o'
merge 'n' + 'e'
merge 'w' + '_'
merge 'lo' + 'we'
merge 'r' + '_'
['lowe', 's', 't', '_']

A real tokeniser

Here's a real tokeniser, from a library called transformers. We give it a sentence. Run it. Look at how the words split. Common ones stay whole, and rare ones break into pieces, marked with hashes. There are special tokens at each end, which mark the start and finish. And the vocabulary has about thirty thousand entries. Each token is just a number, which then looks up an embedding, exactly as in your Tiny GPT.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")

text = "Tokenisation is surprisingly unbelievable."
ids = tok(text)["input_ids"]

print(tok.convert_ids_to_tokens(ids))
print(ids)
print("Vocabulary size:", tok.vocab_size)

['[CLS]', 'token', '##isation', 'is', 'surprisingly', 'unbelievable', '.', '[SEP]']
[101, 19204, 6648, 2003, 10889, 23653, 1012, 102]
Vocabulary size: 30522

Summary