Which Meaning?

A customer writes to Scribe, saying that they sat by the bank. Another says that the bank lost their deposit. It's the same word, with two very different meanings. People handle this without thinking. For decades, it was one of the hardest problems in language processing. So how would you solve it with no training data at all?

• Same word, two meanings
• People don't even notice
• A very old problem

A dictionary and some overlap

In nineteen eighty six, Michael Lesk had a beautifully simple idea. Compare the words around the target word, with the dictionary definition of each meaning. Here are two definitions of bank. Our words function lowercases the text, and it removes common small words. Lesk counts the words that each definition shares with the sentence, and picks the highest. Run it. Money and river are both handled correctly. Lends and customers point to the money meaning. But look at the last line. I went to the bank. There's no overlap at all, and both scores are zero. So it just returned the first meaning, which was a guess. And that's the weakness of the method. It's only as good as the words that happen to overlap.

SENSES = {
    "money": "a financial institution that accepts deposits and lends money to customers",
    "river": "the sloping land beside a river or stream of water",
}
STOP = {"a", "the", "and", "that", "to", "of", "or", "i", "my", "on", "at", "we"}

def words(text):
    return {w.strip(".,").lower() for w in text.split()} - STOP

def lesk(sentence):
    context = words(sentence)
    scores = {s: len(context & words(d)) for s, d in SENSES.items()}
    return max(scores, key=scores.get), scores

for s in ["I deposited money at the bank",
          "We sat on the bank of the river",
          "The bank lends to customers",
          "I went to the bank"]:
    print(s, "->", *lesk(s))

I deposited money at the bank -> money {'money': 1, 'river': 0}
We sat on the bank of the river -> river {'money': 0, 'river': 1}
The bank lends to customers -> money {'money': 2, 'river': 0}
I went to the bank -> money {'money': 0, 'river': 0}

From overlap to context

Word sense has moved through four stages. Hand built dictionaries came first. Then, statistical models learned from labelled examples. Then word vectors, which had a problem, because bank only got one vector, which blended both meanings. Transformers fixed that. Each use of bank gets its own vector, shaped by the words around it.

1. DictionariesLesk overlap
2. Statisticslabelled examples
3. Word vectorsone per word
4. Transformersone per use

Where this still matters

You might think that this is solved, but it comes up all the time. In search, apple the fruit and Apple the company shouldn't mix. In translation, one English word might need two different words in Spanish. In RAG, a chunk about Python snakes can match a question about coding. So test with ambiguous words on purpose, because that's where systems quietly fail.

Summary