In detail
Split the system before you score it
First check whether the required information exists in the source collection. Then evaluate the two RAG stages — retrieve, then generate — and they break for different reasons. If you only measure the final answer, you can’t tell whether a bad response came from fetching the wrong documents or from a model that ignored good ones. Evaluate the two halves separately so you know which one to fix.
Measuring retrieval
The question for retrieval is simple: did the right context come back? Context relevance checks whether the retrieved chunks bear on the query; recall checks whether the documents you needed were found at all. Weak retrieval poisons everything downstream — the best model can’t answer from context it never received.
Twilio’s Vinnie Giarrusso describes this triage on Twilio’s agent platform in episode 7 at 10:00. A basic RAG pipeline is quick to stand up, he says, but problems surface as knowledge sources, use cases and users’ unanticipated questions pile up. When an answer fails, his team reads the trace in order: did the agent pick the correct knowledge source, and if it did, did the top-K chunks set it up to answer? He reports that context-aware retrieval raised Twilio’s retrieval accuracy, without giving numbers. Galileo sponsored Season 1, when Conor worked there, and Giarrusso names Galileo as Twilio’s monitoring tool at 14:00.
Measuring generation
Given the retrieved context, two things matter. Faithfulness asks whether every claim in the answer is supported by the sources, with nothing invented — this is the anti-hallucination check. Answer relevance asks whether the response addresses the user’s question rather than reciting related-but-off material. An answer can be faithful to its sources and still miss the question.
Putting it together
Track retrieval and generation metrics side by side, on your own data, and re-run them on every change. When quality drops, the split tells you immediately whether to tune the retriever or the prompt — instead of guessing at a single number that went down.
Diagnose a policy-answer failure
Worked example, with an invented policy: “Can I claim expenses after 30 days?” The authoritative document says ordinary claims close after 30 days and approved exceptions require finance review. Inspect the exact retrieved chunks before changing the model.
| What you observe | Likely place to investigate | Next experiment |
|---|---|---|
| The policy is absent from the source collection | Coverage | Add or update the authoritative document |
| The document exists but is not retrieved | Retrieval | Test the query, filters and ranking against labeled relevant passages |
| The deadline arrives without the exception clause | Chunking or context assembly | Retrieve the linked clause or change the chunk boundary |
| Both clauses arrive, but the answer invents automatic approval | Generation | Test groundedness and the required qualification |
| The assistant cites a superseded policy accurately | Freshness | Apply effective-date and authority rules before generation |
The table is a diagnostic aid, not evidence that one failure dominates production. Ragas supplies a research basis for scoring retrieval context and generated answers separately. Its automated scores are signals to validate on your own cases, not proof that a policy answer is correct.
Include questions the documents cannot answer
Add a case whose answer is genuinely missing and decide what a useful abstention or escalation looks like. A system that always produces a confident answer can improve apparent answer coverage while making accuracy worse. Conversely, a system that abstains on everything should fail answerable cases. Store the relevant passages or expected missing information with each case so you can inspect the diagnosis.
Conor’s bad-context essay argues for supplying useful context instead of flooding the window. Test that principle on your own retrieval traces. Continue with context and memory engineering when the documents are right but the assembled context is still failing.
Try it: where did the RAG answer break?
Inspect the evidence and answer in each case. Decide what you would investigate first, then reveal the diagnosis.
Question: Can I claim expenses after 30 days?
Current policy: Ordinary expense claims must be submitted within 30 days. Late claims require finance review; approval is not guaranteed.
Invented policy and answers for practice. These are hand-authored examples, with no live model or automated scores.
-
Case 1 A missing policy
- In the collection
- The collection contains travel-booking instructions, but no expense-claim policy.
- Sent to the model
- Book travel through the company travel portal.
- The answer
- These documents do not say whether late expense claims are allowed. Ask finance for the current policy.
Reveal diagnosis for case 1
- Source collection Investigate here
- Retrieved context
- Generated answer
Start with source coverage. The required policy is absent; a different retriever cannot find it in this collection. The answer correctly acknowledges the missing evidence.
Next experiment: Add the authoritative policy, then repeat this question and a question whose answer is still absent. Check that the assistant answers the first and escalates the second.
Context relevance -
Case 2 Half the policy
- In the collection
- The current policy contains both the ordinary deadline and the late-claim exception below.
- Sent to the model
- Ordinary expense claims must be submitted within 30 days.
- The answer
- The ordinary expense-claim deadline is 30 days.
Reveal diagnosis for case 2
- Source collection
- Retrieved context Investigate here
- Generated answer
The exception never reached the model. The answer is supported by its context, but leaves the late-claim question unresolved. Faithfulness alone would miss the incomplete evidence.
Next experiment: Inspect chunk boundaries, filters, ranking and context assembly. Add the exception clause to the supplied context and check whether the answer now explains the finance-review requirement.
Precision and recall -
Case 3 An invented approval
- In the collection
- The current policy contains both clauses below.
- Sent to the model
- Ordinary expense claims must be submitted within 30 days. Late claims require finance review; approval is not guaranteed.
- The answer
- Yes. Late claims are automatically approved.
Reveal diagnosis for case 3
- Source collection
- Retrieved context
- Generated answer Investigate here
The needed evidence arrived, but the answer contradicts it. Investigate generation: automatic approval has no support in the retrieved policy.
Next experiment: Test a response that includes the finance-review requirement and does not promise approval. Keep the retrieved context fixed while comparing prompts or models.
Faithfulness -
Case 4 A convincing old answer
- In the collection
- Both versions exist: an archived 60-day policy and the current 30-day policy with finance review for late claims.
- Sent to the model
- Expense claims may be submitted within 60 days.
- The answer
- Yes. You have 60 days to submit a claim.
Reveal diagnosis for case 4
- Source collection
- Retrieved context Investigate here
- Generated answer
The answer follows the retrieved passage, but that passage is superseded. Its archive status was not sent to the model. A faithfulness check can pass while the answer gives the wrong current guidance.
Next experiment: Check effective dates and document authority during retrieval and context assembly. Repeat the case with both versions still in the collection; verify that the current policy governs the answer.
Faithfulness and its limits -
Case 5 The complete answer
- In the collection
- The current policy contains both clauses below.
- Sent to the model
- Ordinary expense claims must be submitted within 30 days. Late claims require finance review; approval is not guaranteed.
- The answer
- A claim after 30 days requires finance review. Approval is not guaranteed.
Reveal diagnosis for case 5
- Source collection
- Retrieved context
- Generated answer
No defect is shown in this trace. The current evidence covers the question, and the answer preserves the exception and its limit. One passing case does not certify the whole system.
Next experiment: Keep this as an answerable control alongside the missing-policy case. A change that makes the assistant refuse every question should fail this control.
Answer relevance
These are starting points for investigation; real failures can span several stages. Compare the definitions in the Ragas paper and its faithfulness documentation. For missing evidence, read the context-recall documentation. Validate automated judgments against cases you have checked yourself.
Hear it from the guest
“So what we're trying to evaluate is whether, the context we retrieve is, relevant and correct to the questions. And second, based on this context, can the answers be, relevant and correct”
“The ChipNeMo example at NVIDIA, we have seen a lot of when at the beginning, when the co pilot was failing, is that the answer simply doesn't exist in the database. So we need to go back and improve the data instead of the model system.”
“Did it even pick the correct knowledge source, first of all? And if it picked the correct knowledge source, like what did the chunks look like?”
Quotes lightly edited to remove filler words.
Go deeper
- Ragas: Automated Evaluation of Retrieval Augmented Generation Introduces separate evaluation dimensions for retrieval context and generated answers; model scores still need validation.
From the conversation
This explainer is drawn from these episodes — each carries its full transcript.
Concepts in this explainer
FaithfulnessAnswer RelevanceContext RelevanceChunkingHybrid SearchRerankingRetrieval-Augmented Generation (RAG)Accuracy