During pre-training, a model reads text drawn from the web, books, code and other sources and learns to predict the next token at every position. Nobody writes an answer for each example, because the text itself supplies the targets, though people still choose, filter and prepare that text. Over trillions of tokens, the model picks up grammar, facts, styles and reasoning patterns. Pre-training a frontier model takes weeks to months on thousands of specialized chips, which is why relatively few organizations do it from scratch.
What comes out is a base model: good at continuing text, not yet good at following instructions. Post-training turns it into an assistant. Some companies take an open base model and run continuous pre-training on their own domain’s text before post-training it. Pre-training is also where some hallucinations start, since rare facts are hard to learn from patterns alone; see why LLMs hallucinate.
Two ways to turn text into training targets
Next-token prediction is the objective behind GPT-style models, but it is not the only one. Take the sentence “The package arrived on Tuesday morning”:
| Objective | Input the model sees | Target |
|---|---|---|
| Next-token (GPT-style) | The package arrived on | Tuesday |
| Masked-token (BERT) | The package arrived on [MASK] morning | Tuesday |
The masked version also sees the word after the gap; the next-token version never does, which is what lets it generate text one token at a time. Real training uses the tokenizer’s pieces, not whole words. Either way, pre-training happens before you use the model: an ordinary chat request runs inference on fixed weights, and pasting a document into the context is not training on it.
Sources
- Llama Team, Meta: The Llama 3 Herd of Models — Reports pre-training the 405B-parameter flagship on 15.6T text tokens using up to 16K H100 GPUs, then a separate post-training stage; checked October 25, 2026.
- Brown et al.: Language Models are Few-Shot Learners — Describes GPT-3 as an autoregressive language model pre-trained to predict text, which can then perform tasks from prompts alone; checked October 25, 2026.
- Google Research: BERT — BERT is pre-trained by masking some input tokens and predicting them from the context on both sides, a different objective from next-token prediction. Checked November 6, 2026.
Go deeper
From the conversation
-
Thomson 1: The New $40M Legal AI Model | Thomson Reuters Joel Hron -
Agents Are Overtaking Humans in Search: Why Search Must Be Rebuilt -
Most of the Web Will Never Get APIs for AI Agents | Dhruv Batra, Yutori -
The Critical Infrastructure Behind the AI Boom | Cisco CPO Jeetu Patel -
Beyond Transformers: How Liquid AI Is Rethinking LLM Architecture | Maxime Labonne -
First Code, Then AGI: Software’s Event Horizon with Poolside Founders Jason Warner & Eiso Kant