Named Entity Recognition

A support agent reads a message, and instantly picks out the customer's name, the city, the book, and the order number. A computer sees only a line of characters. Named entity recognition gives it that same skill, and it's the first step in turning messy text into a clean database.

• Agents spot names instantly
• Computers see characters
• Text becomes a database

Entities with spaCy

SpaCy comes with a trained entity recogniser. We give it a sentence with people, places, money, dates, and companies. Each entity has a label. Run it. Sam Lee is a person. Toronto and London are places. Monday's a date, and twenty four dollars is money. It spotted all of these from the context, and the same word could be a person in one sentence, and a company in another.

import spacy

nlp = spacy.load("en_core_web_sm")
text = ("Sam Lee from Toronto ordered 'Dune' on Monday for $24. "
        "Amazon quoted three days, but Bright Books sent it from London.")

for ent in nlp(text).ents:
    print(f"{ent.text:<15} {ent.label_:<8} {spacy.explain(ent.label_)}")

Sam Lee         PERSON   People, including fictional
Toronto         GPE      Countries, cities, states
Monday          DATE     Absolute or relative dates or periods
24              MONEY    Monetary values, including unit
Amazon          ORG      Companies, agencies, institutions, etc.
three days      DATE     Absolute or relative dates or periods
London          GPE      Countries, cities, states

Adding your own entities

SpaCy doesn't know about Bright Books' order numbers or products. We can add them, using an entity ruler, placed in front of the statistical model. We give it patterns, one using a regular expression. Run it. It finds the order number and the product, and the place, Oslo, is still found by the normal model. Mixing fixed rules with learned models is a very powerful habit. Use rules for what's exact, and models for what's fuzzy.

import spacy
from spacy.pipeline import EntityRuler

nlp = spacy.load("en_core_web_sm")
ruler = nlp.add_pipe("entity_ruler", before="ner")
ruler.add_patterns([
    {"label": "ORDER", "pattern": [{"TEXT": {"REGEX": "^#\\d{5}$"}}]},
    {"label": "PRODUCT", "pattern": "Bright Plus"},
])

doc = nlp("Order #48213 for Bright Plus never reached Sam in Oslo.")
print([(e.text, e.label_) for e in doc.ents])

[('Bright Plus', 'PRODUCT'), ('Sam', 'PERSON'), ('Oslo', 'GPE')]

How models tag spans

How does a model mark a span that's two words long? The BIO scheme. B means the beginning of an entity, I means inside it, and O means outside. So NER becomes a label for every token, which is just classification, one token at a time. We walk over the tags, and rebuild the spans. Run it. Sam Lee and New York are stitched back together. This same format is used to train transformers for extraction.

tokens = ["Sam", "Lee", "lives", "in", "New", "York"]
tags   = ["B-PER", "I-PER", "O", "O", "B-LOC", "I-LOC"]

for token, tag in zip(tokens, tags):
    print(f"{token:<6} {tag}")

spans, current = [], []
for token, tag in zip(tokens, tags):
    if tag.startswith("B-"):
        current = [token]
        spans.append((tag[2:], current))
    elif tag.startswith("I-"):
        current.append(token)

print([(label, " ".join(words)) for label, words in spans])

Sam    B-PER
Lee    I-PER
lives  O
in     O
New    B-LOC
York   I-LOC
[('PER', 'Sam Lee'), ('LOC', 'New York')]

Summary