BERTScore

BERTScore was introduced in 2019 by Tianyi Zhang and colleagues at Cornell and ASAPP as an answer to the main flaw in BLEU and ROUGE: they match surface words, so a paraphrase that means the same thing scores poorly (Zhang et al., BERTScore: Evaluating Text Generation with BERT). Instead of counting exact matches, BERTScore embeds every token in the candidate and the reference with a pretrained contextual model, matches each token to its most similar counterpart by cosine similarity, and reports precision, recall and F1 from those matches. Because the embeddings are contextual, each token is represented in light of the sentence around it.

The authors tested it on the outputs of 363 machine translation and image captioning systems and found it correlated better with human judgments than the existing metrics, and selected better models. The metric also supports optional importance weighting, estimated from corpus statistics, so rare and informative words count for more than common ones; check whether a reported score used it before comparing numbers.

The tradeoffs are practical. Scores depend on which model and which layer computed the embeddings, so numbers are only comparable when that setup is fixed. The reference implementation prints a hash such as roberta-large_L17_no-idf_version=0.3.0 that records the model, layer, weighting and version for that reason, and its optional baseline rescaling turns an F1 of 0.959 into 0.759 on the same example input files (bert_score on GitHub). It still needs a reference text, and semantic similarity is not correctness: an answer that is close in meaning to the reference but wrong on a number or a name can score well. For generated answers, pair it with a faithfulness check against the source documents or an LLM-as-a-judge rubric, and validate either against human labels. LLM-as-a-judge vs human evaluation covers that calibration step.