Start with a phone keyboard that suggests the next word. An autoregressive language model does a far more capable version of the same job. It reads the text so far, scores every token in its vocabulary by how likely it is to come next, picks one, appends it and runs again. A three-paragraph answer is hundreds of those small steps in a row. Settings such as temperature control how adventurous each pick is.
Training teaches the scoring. During pre-training, the model sees huge amounts of text with the next token hidden, guesses it, and is corrected each time it is wrong. Later steps such as RLHF tune which continuations it prefers, so it answers questions and follows instructions instead of just continuing a web page. The InstructGPT paper makes that point directly: predicting the next token of internet text is not the same goal as following a user’s instructions.
Why it matters: the model is optimized to produce a likely continuation, not to check facts. That is one reason a fluent answer can still be wrong (see AI hallucination). It also explains why the wording and context you give a model change its answer so much.
On the show, guests use the idea to explain both limits and strategy. In episode 54 at 24:24, Dan Klein argues that systems in “a next token prediction mode” lack a sense of what they do and don’t know; they emit tokens based on “contextual plausibility.” In episode 55 at 2:03, Neo4j’s Sudhir Hasbe says generative technology “was about how do you predict the next word,” and argues enterprises need their own knowledge alongside the model. Galileo, then the show’s presenting sponsor, is thanked in the episode 54 opening and the episode 55 opening.
In episode 23 at 3:29, Poolside’s Eiso Kant describes the view that you could “just scale up next token prediction” to reach AGI, and says Poolside bet instead on scaling reinforcement learning. That is his company’s strategy, not a settled result. Episode 23 aired in season two, which Galileo sponsored while Conor worked there (Conor mentions Galileo at 19:01).
Hear it from the guest
“Systems don't really, do this when they're in like a next token prediction mode.”
“All we have to do is just scale up next token prediction, scale up model size, provide more data”
Quotes lightly edited to remove filler words.
Sources
- Brown et al.: Language Models are Few-Shot Learners (GPT-3) — Describes GPT-3 as an autoregressive language model with 175 billion parameters, applied to tasks through text prompts without gradient updates; checked October 24, 2026.
- Ouyang et al.: Training language models to follow instructions with human feedback (InstructGPT) — Explains that predicting the next token of internet text is a different objective from following a user's instructions, and adds labeler demonstrations and rankings to close the gap; checked October 24, 2026.
- Nie et al.: Large Language Diffusion Models (LLaDA) — Introduces a diffusion language model that predicts masked tokens through a reverse generation process, showing that next-token generation is not universal to language models; checked October 24, 2026.
Go deeper
From the conversation
-
Why LLMs Are Plausibility Engines, Not Truth Engines | Dan Klein, Scaled Cognition -
First Code, Then AGI: Software’s Event Horizon with Poolside Founders Jason Warner & Eiso Kant -
Hallucinations Are a Data Architecture Problem | Sudhir Hasbe, Neo4j -
Vercel's Playbook for AI Agents: From Vibe Check to Production | Malte Ubl