RAG can fail in two places, and context relevance isolates the first: retrieval. It asks whether the passages the system pulled back are about the question, independent of what the model later does with them. The Ragas framework defines retrieved context as relevant to the extent that it contains only the information needed to answer, and scores it by having an LLM extract the sentences from the context that help answer the question, then dividing the number extracted by the total number of sentences (Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation). A low score means the model is reading mostly noise.
The Ragas authors give two reasons this matters beyond accuracy. Long context passages cost money to send to a model, and models are often less effective at using information buried in the middle of a long context, a finding they cite from Liu et al.’s work on long-context retrieval. So retrieving more is not a free safety margin; padding the context with marginal passages can lower answer quality.
Measure it separately from the answer. If answers are bad, context relevance tells you whether to fix the retriever (chunking, embeddings, query rewriting, reranking) or the generation step. The Stanford ARES framework treats context relevance as its own dimension alongside answer faithfulness and answer relevance for the same reason (Saad-Falcon et al., ARES). Ranking metrics such as mean reciprocal rank complement it when you have labeled answers: MRR tells you where the right passage landed, and context relevance tells you how much irrelevant text came with it.
Pair it with answer relevance and faithfulness for the full picture. How to evaluate a RAG system covers the setup, and vector database vs knowledge graph covers retrieval design choices that move this score.
Sources
- Es et al.: Ragas: Automated Evaluation of Retrieval Augmented Generation — Defines context relevance and scores it as the share of context sentences an LLM extracts as needed to answer.
- Saad-Falcon et al.: ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems — Treats context relevance as its own dimension beside answer faithfulness and answer relevance.