Normalising Text

A customer writes, I LOVE this book. Another writes, i love this book!! A third writes, loved it. To you, these are close to the same message. To a simple program, they don't match at all. Normalisation brings different spellings of the same thing together.

• I LOVE this book
• i love this book!!
• loved it

A cleaning function

Here's a reusable cleaning function. Unicode normalise fixes strange characters, like the wide letters at the end of our test sentence. Lower makes everything lower case. We swap web links for a marker, since each link is unique, but the fact that there is a link might matter. And we collapse spaces. Run it. Messy input, and neat output. A warning, though. Modern transformer models are trained on natural text, so you should do much less cleaning for those.

import re
import unicodedata

def clean(text):
    text = unicodedata.normalize("NFKC", text)
    text = text.lower()
    text = re.sub(r"https?://\S+", " [URL] ", text)
    text = re.sub(r"\s+", " ", text).strip()
    return text

print(clean("  CHECK   this: https://bit.ly/x   ＦＵＬＬ width!  "))

check this: [URL] full width!

Stemming and lemmas

Stemming chops off word endings, with crude rules. Lemmatising uses a dictionary, to find the true base form. Here we compare the two. Run it. The stemmer turns studies into studi, which isn't a word. The lemmatiser gives study, and it knows that mice is the plural of mouse, and that were goes to be. Stemming is fast and rough. Lemmatising is slower, but much more accurate.

import spacy
from nltk.stem import PorterStemmer

nlp = spacy.load("en_core_web_sm")
stem = PorterStemmer()

words = "the mice were running and studies better than leaves"
for t in nlp(words):
    if not t.is_stop:
        print(f"{t.text:<9} stem: {stem.stem(t.text):<8} lemma: {t.lemma_}")

mice      stem: mice     lemma: mouse
running   stem: run      lemma: run
studies   stem: studi    lemma: study
better    stem: better   lemma: well
leaves    stem: leav     lemma: leaf

Stop words, and a trap

Stop words are very common words, such as the, and, and is. Old systems removed them to save space, and to focus on useful words. Run it. But look at what's left of our sentences. The book was not good becomes, book good. We've just reversed the meaning! Not and never are stop words. So never remove stop words without thinking, especially for sentiment.

import spacy

nlp = spacy.load("en_core_web_sm")
print("Stop words known:", len(nlp.Defaults.stop_words))

for text in ["The book was not good", "I would never buy this"]:
    kept = [t.text for t in nlp(text) if not t.is_stop]
    print(f"{text:<26} -> {kept}")

Stop words known: 326
The book was not good      -> ['book', 'good']
I would never buy this     -> ['buy']

Summary