BERT

In twenty nineteen, Google announced one of the biggest leaps in the history of its search engine. For the first time, it could understand the small words, like to and from, that change a whole query. The model behind that was called BERT. It reads in both directions, all at once, and it's a landmark in language AI.

• A giant leap for Google Search
• Understanding small words
• BERT reads both ways at once

BERT fills in the blank

BERT was trained by hiding words, and predicting them. So we can ask it directly. Fill mask gives it a sentence with a blank. Run it. For, the book was very, it suggests words like good and popular. For, my parcel never, it suggests arrived and came. It knows what fits, from reading billions of words. That's not looking up a dictionary. It's a real sense of language.

from transformers import pipeline

fill = pipeline("fill-mask", model="distilbert-base-uncased", device="cpu")

for text in ["The book was very [MASK].", "My parcel never [MASK]."]:
    print(text)
    for r in fill(text)[:3]:
        print(f"   {r['token_str']:<10} {r['score']:.2f}")

The book was very [MASK].
   popular    0.30
   successful 0.11
   controversial 0.05
My parcel never [MASK].
   materialized 0.09
   happened   0.04
   changed    0.03

Meaning depends on context

Here's what makes this so powerful. In word two vec, bank has one vector, forever. In BERT, a word's vector depends on the sentence around it. We compare bank in three sentences, two about rivers, and one about money. Run it. The two river banks are much closer to each other, than to the money bank. Same word, different vectors. BERT has solved the ambiguity that we met in lesson one.

import torch
from transformers import AutoModel, AutoTokenizer

name = "distilbert-base-uncased"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name).eval()

def vector(text, word):
    enc = tok(text, return_tensors="pt")
    with torch.no_grad():
        out = model(**enc).last_hidden_state[0]
    i = enc["input_ids"][0].tolist().index(tok.convert_tokens_to_ids(word))
    return out[i]

a = vector("We sat on the river bank.", "bank")
b = vector("The boat drifted to the muddy bank.", "bank")
c = vector("I paid at the bank today.", "bank")

sim = torch.nn.functional.cosine_similarity
print("river vs muddy shore:", round(sim(a, b, dim=0).item(), 3))
print("river vs money bank: ", round(sim(a, c, dim=0).item(), 3))

river vs muddy shore: 0.874
river vs money bank:  0.685

[transformers] DistilBertModel LOAD REPORT from: distilbert-base-uncased
Key                     | Status     |  |
------------------------+------------+--+-
vocab_layer_norm.weight | UNEXPECTED |  |
vocab_transform.bias    | UNEXPECTED |  |
vocab_transform.weight  | UNEXPECTED |  |
vocab_layer_norm.bias   | UNEXPECTED |  |
vocab_projector.bias    | UNEXPECTED |  |

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.

Summary