In detail
Check the claim, not the confidence
An answer can be fluent, relevant and unsupported. Start with the claim that changes a decision: an amount, deadline, eligibility condition or reported action. Find the actual passage or record and compare what it says with the answer. Do this before asking whether the answer sounds professional.
The reading-list case Mikiko Chandrasekhar mentions in episode 33 at 10:20 shows what skipping that step costs. In May 2025 a syndicated summer reading list in the Chicago Sun-Times recommended books that do not exist. The paper’s publisher said the list was created with an AI tool by a freelancer working for its distributor, King Features, and ran without review from the paper’s editorial team. At least one other paper, The Philadelphia Inquirer, ran the same list. Each fake title was a checkable claim nobody checked.
Worked example, with invented facts: policy P-17 allows refunds within 30 days. After that, an approved exception is required. The assistant replies, “You are eligible for an automatic refund through day 60,” and cites P-17.
| Answer claim | Evidence to inspect | Verdict in this example |
|---|---|---|
| The deadline is 60 days | The current policy’s deadline clause | Unsupported: the clause says 30 |
| Exceptions are automatic | The exception-approval clause | Unsupported: approval is required |
| The refund has been issued | The refund ledger | Unknown until the record is checked |
The citation is real. The interpretation is wrong. Keeping those failures separate tells you what to fix.
In episode 5 at 30:00, Chip Huyen separates whether the retrieved context is relevant and correct from whether the answer based on it is relevant and correct. The RAG evaluation explainer applies that distinction: receiving the right policy and using it correctly are separate checks.
Check scope and dates
A policy for one region may not apply to another. An old page can accurately describe a policy that has since changed. A transcript records what a guest said at the time; it does not establish that a vendor still uses the same model today. Ask which source has authority for this question, when it took effect, and whether its qualifications survive in the answer.
For a calculation, recompute the result from the supplied inputs. For an action, inspect the system of record. “I sent the message” needs a sent-message record; a draft or an accepted tool request is a different state. These are suggested verification steps, not claims that one source can settle every decision.
Use another model as a helper
A second model can identify claims to check or compare a passage with an answer. It can also repeat the first model’s mistake. Hamel Husain’s discussion in episode 37 connects useful evaluation to reading actual failures (8:23) and comparing judge decisions with human labels (15:38). The judge explainer turns that into a calibration process.
For an individual answer, ask the helper to identify the supporting passage and any missing evidence. Read those passages yourself when the decision matters. For a repeated workflow, keep reviewed cases and track false approvals; a high agreement score can hide the very mistakes you need to catch.
Check yourself
An answer quotes the policy accurately, then applies it to the wrong customer category. Is it grounded? The quotation is supported, but the conclusion is not established for that customer. Check applicability as well as wording. Continue with RAG evaluation to diagnose whether the wrong context was retrieved or the right context was misused.
Disclosure: Episodes 5, 33 and 37 were sponsored by Galileo while Conor worked there.
Hear it from the guest
“A few of those summer reading lists that came out of reputable sites and media channels were fake. The books did not exist.”
“If you can trace back to the exact words and draw a box around the words that a legal contract came from when an AI agent generates a response, that's a very powerful tool”
Quotes lightly edited to remove filler words.
Go deeper
- Using LLM-as-a-Judge for Evaluation Shows how human judgments and observed failures shape a useful rubric rather than relying on a generic model score.
From the conversation
This explainer is drawn from these episodes — each carries its full transcript.
Concepts in this explainer
Answer EngineDeep ResearchSelf-CorrectionSycophancyRetrieval-Augmented Generation (RAG)AI EvaluationGenerative AILLM as a Judge