Answer Relevance

An answer can be grounded in good sources and still miss the point: it can respond to a nearby question, bury the answer in caveats, or summarize the documents instead of answering. Answer relevance scores that directly. The Ragas evaluation framework defines it as whether the generated answer addresses the question that was asked, and says explicitly that it does not judge factuality; it penalizes answers that are incomplete or padded with redundant information (Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation).

The Ragas method is a useful trick to understand even if you use a different tool. An LLM reads the answer and writes several questions that the answer would fit. Each generated question is embedded and compared with the user’s original question by cosine similarity, and the score is the average. If the answer drifted, the questions it implies drift too, and the similarity drops.

Answer relevance is one of three scores that RAG evaluation usually separates. Context relevance checks whether retrieval pulled the right material, and faithfulness checks whether the answer’s claims are supported by that material. ARES, a later framework from Stanford, uses the same three dimensions and trains small judge models for each, then corrects their errors with a few hundred human-labeled examples (Saad-Falcon et al., ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems). Keeping the scores apart is what makes them useful for debugging: a relevant but unfaithful answer points at the generator, and a faithful answer to the wrong question often points at the prompt or at retrieval.

Because these scores come from model judges, check them against human labels on a sample before you rely on them, for example by measuring agreement with Cohen’s kappa. How to evaluate a RAG system walks through the full setup.