In detail
1. For creative work, hallucination is the feature
When you want a poem, a name, a fresh angle, the model inventing something outside its training is exactly what you came for. Chip Huyen is blunt about it in episode 5 at 30:00: hallucinations are going to be really great for creative use cases. Databricks’ Denny Lee goes further in episode 22 at 11:54: a model “has to hallucinate a little bit” to be creative; with no hallucination at all, he says, you would just have search. Dan Klein adds in episode 54 at 9:02 that for many generative models, such as image generation, “the hallucination is actually the product.” There, the invention is the goal.
2. For factual work, it is the bane of your existence
The moment the task depends on being correct, the same behavior becomes the risk you spend all your time fighting. A hallucination is bad specifically when it is factually inconsistent with what is supposed to be true. Most enterprise use cases live here. Part of the reason models guess instead of saying they don’t know is how they’re graded: Kalai and colleagues argue in Why Language Models Hallucinate that training and evaluation reward guessing over admitting uncertainty. For factual work, an eval that gives credit for an honest “I don’t know” is worth building. Why do LLMs hallucinate? covers the mechanism.
3. The dangerous one: looks right, isn’t
Vivienne Zhang’s example in episode 5 at 33:00, which she credits to Jensen Huang at GTC, sticks. Ask an NVIDIA copilot “what is CTL” and it returns a generic, textbook expansion of the acronym. But for a chip designer inside NVIDIA the right answer is the internal one, “Compute Trace Library.” The response is confident, plausible, and wrong in context. For a new employee, that is just as insidious as a completely wrong answer, because it looks right.
Why it matters
“Stop the model from hallucinating” is the wrong goal. The right goal is to know which mode you are in. If you are doing creative work, let it run. If you are doing factual work, your real job is grounding it and catching the answers that look right but are not. Are hallucinations a data problem? covers where that grounding comes from.
Disclosure: Episode 5 was sponsored by Galileo while Conor worked there (read at 20:00). Episode 22 aired while Chain of Thought was Galileo’s podcast and Conor worked there (42:47); Databricks, the guest’s employer, is an investor in and partner of Galileo (13:44). Episode 54 thanks Galileo as its Season 3 presenting sponsor in the opening read.
Hear it from the guest
“So I think hallucinations is going to be really, really great things for creative use cases. It's a bad thing for like use cases that depend on factual consistency.”
“The application gives an answer that looks correct, but it's not what the user is looking for, and I will consider that a hallucination.”
“In many cases, a generative model, the hallucination is actually the product.”
“It has to hallucinate a little bit to have that form of creativity.”
Quotes lightly edited to remove filler words.
From the conversation
This explainer is drawn from these episodes — each carries its full transcript.
Concepts in this explainer
AI HallucinationGroundingLarge Language ModelAI EvaluationGenerative AIText-to-Image