Scoring Translations

A translation system says that the book arrived on Monday. Another says Monday was when my novel got here. Both are correct. So how could a computer score them? For years, translation research depended on one simple number, and you need to know its strengths and its flaws.

• Two correct translations
• How do you score them?
• One simple number

BLEU

Bleu compares a candidate with a reference translation. It counts how many single words and pairs of words match. And it applies a penalty if the candidate's too short, or a system could just write one safe word. Run it. A perfect copy scores one. A close one gets point six three. But look at this. Monday was when my novel got here means the same thing, and it scores zero. That's the flaw. It rewards matching words, not meaning. And the short answer is correctly punished. So use it to compare systems on large test sets, and never to judge one sentence.

from collections import Counter
import math

def bleu(candidate, reference, n=2):
    c, r = candidate.split(), reference.split()
    scores = []
    for k in range(1, n + 1):
        cg = Counter(tuple(c[i:i + k]) for i in range(len(c) - k + 1))
        rg = Counter(tuple(r[i:i + k]) for i in range(len(r) - k + 1))
        scores.append(max(sum((cg & rg).values()), 1e-9) / max(sum(cg.values()), 1))
    penalty = min(1, math.exp(1 - len(r) / len(c)))
    return penalty * math.exp(sum(map(math.log, scores)) / n)

ref = "the book arrived on monday"
print(f"Perfect: {bleu('the book arrived on monday', ref):.2f}")
print(f"Close:   {bleu('the book came on monday', ref):.2f}")
print(f"Same meaning: {bleu('monday was when my novel got here', ref):.2f}")
print(f"Too short: {bleu('the book', ref):.2f}")

Perfect: 1.00
Close:   0.63
Same meaning: 0.00
Too short: 0.22

Scores for language

Each task has its own scores. Bleu for translation. Rouge for summaries, which is another overlap score. Word error rate for speech. And for meaning, embedding similarity gets closer. Today, teams often use a large language model as a judge, with a clear rubric, and they check that judge against humans. No score replaces reading the outputs.

Summary