BLEU (Bilingual Evaluation Understudy) came out of IBM Research in 2002 as a cheap stand-in for human review of machine translation (Papineni et al., BLEU: a Method for Automatic Evaluation of Machine Translation). It compares an output with one or more human reference translations using modified n-gram precision: for each n-gram length, the share of the output’s n-grams that appear in a reference, with each match capped at the number of times that n-gram occurs in any single reference. A multiplicative brevity penalty then lowers the score of outputs shorter than the references, since precision alone would reward a translation that says very little.
The original paper reports that BLEU correlates highly with human judgment when scores are averaged over a test corpus, and it says plainly that scores on individual sentences often diverge from human ratings. That distinction is still the right way to use it: as a corpus-level signal for comparing systems on the same test set, not as a grade for one output.
The limits were documented early. Callison-Burch, Osborne and Koehn showed that many different translations can receive the same BLEU score while humans rate them very differently, and gave two real cases (Re-evaluating the Role of BLEU in Machine Translation Research). In the 2005 NIST evaluation, the Arabic-to-English entry that human judges ranked first was ranked sixth by BLEU; that entry was not fully automatic (English speakers chose among machine-translated phrases and post-edited the result), but BLEU still failed to rank it the way people did. In their own experiment, BLEU vastly underestimated the quality of a rule-based system (Systran) compared with statistical systems. Their conclusion was that a higher BLEU score does not necessarily mean a better translation.
For builders, that makes BLEU a regression check for translation or other tasks with fixed reference text. METEOR and BERTScore were built to credit correct wording that differs from the reference, and does AI replace human translators? covers why fluent machine translation can still get facts and cultural context wrong.
Sources
- Papineni et al.: BLEU: a Method for Automatic Evaluation of Machine Translation — The 2002 paper: modified n-gram precision, the brevity penalty, and corpus-level correlation with human judgment.
- Callison-Burch, Osborne and Koehn: Re-evaluating the Role of BLEU in Machine Translation Research — Shows that different translations can share a BLEU score while humans rate them very differently.