Running a Real Model in Code

Let's touch a real model. In this lesson, we will load GPT two, a small open model, on an ordinary computer, and watch it work. You will see the tokenizer, the probabilities, and generated text, all produced for real, as the code runs.

• A real open model
• Tokenizer, chances, and text
• All produced for real

The tokenizer in code

First, the tokenizer. With the transformers library, one line loads the tokenizer for GPT two. We encode a sentence into token I Ds, and then decode each ID back, to see the pieces. This is exactly the step from our lesson on tokens, now running for real, in code.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
ids = tok.encode("Generative AI is amazing")
print(ids)
print([tok.decode([i]) for i in ids])

[8645, 876, 9552, 318, 4998]
['Gener', 'ative', ' AI', ' is', ' amazing']

Next-token chances in code

Now, the model itself. We load GPT two, give it a sentence, and ask for its raw scores for the next token. We turn the scores into probabilities with softmax, and print the five most likely next words. These are real numbers, from a real model, and they show the same prediction game, that powers every assistant.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

ids = tok("The best way to learn is to", return_tensors="pt").input_ids
with torch.no_grad():
    logits = model(ids).logits[0, -1]
probs = torch.softmax(logits, dim=-1)
top = torch.topk(probs, 5)
for p, i in zip(top.values, top.indices):
    print(f"{tok.decode([int(i)]).strip():>10}  {p:.1%}")

        go  5.2%
     learn  5.1%
      take  3.7%
       get  3.3%
        be  3.2%

Generating text in code

Finally, generation. We use the pipeline helper, set a random seed, so the result can be repeated, and ask for twenty new tokens. The model continues our sentence, one token at a time. A small model like this, is not very smart, but it shows the whole mechanism, in a few lines.

from transformers import pipeline, set_seed

set_seed(7)
gen = pipeline("text-generation", model="gpt2", device="cpu")
out = gen("Once upon a time in a small village,", max_new_tokens=20, do_sample=True, temperature=0.8)
print(out[0]["generated_text"])

Once upon a time in a small village, only a few men were there to help the peasants build a home, and to offer their support to

Where to go next

If you enjoyed that, here is where to go. The Hugging Face website hosts hundreds of thousands of open models and datasets, and the transformers library loads them in a line. Try a larger model, with a graphics card, or a quantised one, with Ollama. Change the temperature, and watch the results. Read the model cards, which explain what each model can, and cannot do. Curiosity, and experiments, are the best teachers.

Summary