Phone keyboards that suggest the next word are a simple version of the idea. Pick the suggestion, and the keyboard now uses that word to suggest the one after. A chatbot does the same at much larger scale. When you watch an answer appear word by word, you are watching an autoregressive model at work. Each new token is chosen from a probability distribution conditioned on the prompt and every token already written, then appended to the input for the next step.
In practice. Most current large language models are decoder-only autoregressive models built on the transformer. The Hugging Face documentation calls this causal language modeling: during training, each position may only look at tokens to its left, so the model learns next-token prediction. That differs from masked language models such as BERT, which see text on both sides of a gap. Encoder-decoder models also usually generate autoregressively; what sets them apart is the separate encoder that reads the whole input first, where a decoder-only model handles input and output in one stream.
Costs and side effects. Because tokens come out one after another, longer answers take longer and cost more, which is why vendors report time to first token and price output tokens separately. An early mistake also becomes part of the input for everything that follows, so one wrong token can steer the rest of an answer.
On the show. In episode 54, at 21:21, Dan Klein, a UC Berkeley professor, says that “if all you need is a stream of tokens, then sort of an autoregressive token predictor is great”, but that systems that take actions need more structure. At 31:02 he worries that “token-based autoregressive LLMs” without extra semantic structure end up as “plausibility engines”. Those are Klein’s views. The episode’s sponsor read names Galileo.
Why it matters. The autoregressive design explains much of how LLM products behave: streaming text, per-token pricing, non-repeatable outputs when sampling is on, and errors that compound within a single answer.
Hear it from the guest
“If all you need is a stream of tokens, then sort of an autoregressive token predictor is great”
“Token-based autoregressive LLMs that do not have any additional semantic pairing structure”
Quotes lightly edited to remove filler words.
Sources
- Hugging Face Transformers documentation: Glossary — Glossary that equates autoregressive models with decoder models pretrained by causal language modeling, reading text in order and predicting the next word; checked October 24, 2026
- Hugging Face Transformers documentation: Causal language modeling — Task guide explaining that a causal language model predicts the next token and can only attend to tokens on its left; checked October 24, 2026
