Using Pretrained Transformers

You could spend millions training a language model. Or you could download one, for free, with one line. Hugging Face hosts well over a million models, and it's become the place where the A I world shares its work. Knowing your way around it is a core skill.

• Millions to train one
• Or one line to download
• The GitHub of AI

One line to a model

The pipeline function gets you from nothing to a working model, in a single line. Here's a sentiment model. We pass in two sentences. Run it. It downloads the weights, tokenises the text, runs the model, and gives you a label with a score. There are pipelines for translation, summarising, speech, and many more. This is the fastest way to find out whether a task is even possible.

from transformers import pipeline

clf = pipeline("sentiment-analysis", device="cpu",
               model="distilbert-base-uncased-finetuned-sst-2-english")

for text in ["This course is brilliant.", "The battery died after an hour."]:
    r = clf(text)[0]
    print(f"{r['label']:<9} {r['score']:.3f}  {text}")

POSITIVE  1.000  This course is brilliant.
NEGATIVE  1.000  The battery died after an hour.

Looking inside

For more control, load the parts yourself. The tokeniser turns text into numbers. And the model gives one vector per token. Run it. Six tokens, each with seven hundred and sixty eight numbers, from a model with around twenty six million weights. It's the same design that you built by hand, just bigger. And this hidden output is what you'd attach a new head to.

from transformers import AutoModel, AutoTokenizer
import torch

name = "distilbert-base-uncased"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name)

inputs = tok(["Deep learning is fun"], return_tensors="pt")
with torch.no_grad():
    out = model(**inputs).last_hidden_state

print("Tokens:", inputs["input_ids"].shape[1])
print("Output:", list(out.shape))
print("Weights:", sum(p.numel() for p in model.parameters()))

Tokens: 6
Output: [1, 6, 768]
Weights: 66362880

[transformers] DistilBertModel LOAD REPORT from: distilbert-base-uncased
Key                     | Status     |  |
------------------------+------------+--+-
vocab_layer_norm.bias   | UNEXPECTED |  |
vocab_transform.weight  | UNEXPECTED |  |
vocab_layer_norm.weight | UNEXPECTED |  |
vocab_transform.bias    | UNEXPECTED |  |
vocab_projector.bias    | UNEXPECTED |  |

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.

Summary