Handling Negation

This book is not great. A simple word counter sees great, and scores that as positive. Not bad at all gets scored as negative. Negation is one of the classic traps in language. Counting words can't see how those words relate to each other.

• 'Not great' scores as positive
• 'Not bad' scores as negative
• Word counts miss the relation

A simple flip rule

We add one small rule. After a negation word, flip the sentiment of the next few words, stopping at any punctuation. Run it. This book is not great goes from plus one to minus one. Correct. Not bad at all flips from minus one to plus one. Correct. And in great story, not broken, we now correctly get a positive result. A rule like this was state of the art in the early two thousands. It still fails on things like I wouldn't say it wasn't good. That sort of sentence is exactly why transformers took over, as attention can connect not to the word that it modifies.

import re

POS, NEGW = {"good", "great", "love", "happy"}, {"bad", "awful", "hate", "broken"}
FLIP = {"not", "never", "no", "hardly"}

def simple(text):
    w = text.lower().split()
    return sum(x in POS for x in w) - sum(x in NEGW for x in w)

def with_flip(text):
    score, flip = 0, 0
    for x in re.findall(r"[a-z]+|[.,!?]", text.lower()):
        if x in FLIP:
            flip = 3
        elif x in ".,!?":
            flip = 0
        elif x in POS or x in NEGW:
            s = 1 if x in POS else -1
            score += -s if flip else s
        flip = max(0, flip - 1) if x not in FLIP else flip
    return score

for t in ["this book is great", "this book is not great", "not bad at all", "great story, not broken"]:
    print(f"{t:<26} simple {simple(t):+d}  flip {with_flip(t):+d}")

this book is great         simple +1  flip +1
this book is not great     simple +1  flip -1
not bad at all             simple -1  flip +1
great story, not broken    simple +0  flip +2

The ladder of language understanding

Think of this as a ladder. Word counts see which words are there. N grams see short phrases. Rules can handle known patterns. Embeddings capture word meanings. And transformers relate each word to all of the others. Each rung costs more. So pick the lowest rung that solves your problem.

Summary