ROUGE (Recall-Oriented Understudy for Gisting Evaluation) scores a machine-written summary against one or more summaries written by people. Chin-Yew Lin introduced the package in 2004 and tested it against human judgments from the Document Understanding Conference, the NIST-sponsored summarization evaluation (Lin, ROUGE: A Package for Automatic Evaluation of Summaries). The variants you will see most often count different kinds of overlap. ROUGE-N counts shared n-grams, so ROUGE-1 is single words and ROUGE-2 is word pairs. ROUGE-L uses the longest common subsequence, which rewards words that appear in the same order without requiring them to be adjacent. ROUGE-S counts skip-bigrams, any pair of words in sentence order with gaps allowed.
The “recall-oriented” part is the design choice that separates it from BLEU. BLEU asks how much of the output appears in the reference; ROUGE-N asks how much of the reference the output covered. That fits summarization, where the failure you most want to catch is a summary that leaves out the point. With several references, Lin’s formulation gives more weight to n-grams that appear in more of them, so a summary that matches what most human writers included scores higher.
ROUGE still measures overlap, not meaning. An abstractive summary that rephrases the source can score low, and one that copies sentences out of the source can score high. The SummEval study, which re-evaluated 14 automatic metrics against expert and crowd-sourced ratings of news summaries, found that most metrics correlated only weakly or moderately with human judgments of coherence, which the authors connect to metrics relying on hard or soft subsequence alignment (Fabbri et al., SummEval: Re-evaluating Summarization Evaluation). Use it to track a summarizer across model or prompt changes, where the same references stay fixed and a drop is a signal to look closer. To judge whether a summary is accurate and useful, pair it with a semantic metric such as BERTScore, a faithfulness check against the source, or LLM-as-a-judge scoring calibrated against human review. How to test an AI system covers where each of those fits in an evaluation suite.
Sources
- Lin: ROUGE: A Package for Automatic Evaluation of Summaries — The 2004 paper defining ROUGE-N, ROUGE-L and related variants, tested against DUC human judgments.
- Fabbri et al.: SummEval: Re-evaluating Summarization Evaluation — Re-evaluates 14 automatic metrics; most correlate weakly or moderately with human coherence ratings.