METEOR (Metric for Evaluation of Translation with Explicit ORdering) was proposed by Satanjeev Banerjee and Alon Lavie at Carnegie Mellon in 2005 as a direct response to weaknesses they saw in BLEU (METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments). It aligns words in the output with words in a reference translation, matching exact words, words that share a stem, and synonyms. From that alignment it computes unigram precision and recall and combines them with a harmonic mean that puts most of the weight on recall (the paper uses P and 9R). A fragmentation penalty then lowers the score when matched words are scattered rather than in contiguous runs, by up to half when there are no matching bigrams or longer.
The authors’ argument against BLEU was specific. Its fixed brevity penalty does not make up for the lack of recall, it uses higher-order n-grams as an indirect measure of grammatical well-formedness, and n-gram counting never aligns specific words in the output with words in the reference. They report better sentence-level agreement with human judgments than unigram precision and recall alone on the 2003 LDC TIDES Arabic-to-English and Chinese-to-English data, while noting that sentence-level correlation was still low overall.
METEOR is still a reference-based, surface-level metric. Synonym matching depends on linguistic resources for the language, so coverage varies; the later Meteor Universal release addressed that by extracting paraphrase tables and function-word lists automatically from the bilingual text used to train a translation system, and outperformed BLEU on Russian and Hindi (Denkowski and Lavie, Meteor Universal). A fluent output that is factually wrong can still match a reference well. Use it the same way as BLEU, as a comparison across systems on a fixed test set, and reach for BERTScore or LLM-as-a-judge when outputs are open-ended. Does AI replace human translators? covers why fluent machine translation can still get facts and cultural context wrong.
Sources
- Banerjee and Lavie: METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments — The 2005 paper: exact, stem and synonym alignment, a recall-weighted harmonic mean and a fragmentation penalty.
- Denkowski and Lavie: Meteor Universal: Language Specific Translation Evaluation for Any Target Language — Extracts paraphrase tables and function-word lists automatically to extend METEOR to more languages.
