Skip to content

AI Glossary · Concepts

What is Pre-training?

Pre-training

Pre-training is the first phase of building a language model, and usually the most expensive, in which it learns general patterns of language and knowledge by predicting the next token across a huge body of text. The result is a base model.

Also known as: pretraining

· Updated · Chain of Thought

Model Training

  • 2 explainers
  • 6 episodes
  • 3 sources

During pre-training, a model reads text drawn from the web, books, code and other sources and learns to predict the next token at every position. Nobody writes an answer for each example, because the text itself supplies the targets, though people still choose, filter and prepare that text. Over trillions of tokens, the model picks up grammar, facts, styles and reasoning patterns. Pre-training a frontier model takes weeks to months on thousands of specialized chips, which is why relatively few organizations do it from scratch.

What comes out is a base model: good at continuing text, not yet good at following instructions. Post-training turns it into an assistant. Some companies take an open base model and run continuous pre-training on their own domain’s text before post-training it. Pre-training is also where some hallucinations start, since rare facts are hard to learn from patterns alone; see why LLMs hallucinate.

Two ways to turn text into training targets

Next-token prediction is the objective behind GPT-style models, but it is not the only one. Take the sentence “The package arrived on Tuesday morning”:

ObjectiveInput the model seesTarget
Next-token (GPT-style)The package arrived onTuesday
Masked-token (BERT)The package arrived on [MASK] morningTuesday

The masked version also sees the word after the gap; the next-token version never does, which is what lets it generate text one token at a time. Real training uses the tokenizer’s pieces, not whole words. Either way, pre-training happens before you use the model: an ordinary chat request runs inference on fixed weights, and pasting a document into the context is not training on it.

Sources

  • Llama Team, Meta: The Llama 3 Herd of Models — Reports pre-training the 405B-parameter flagship on 15.6T text tokens using up to 16K H100 GPUs, then a separate post-training stage; checked October 25, 2026.
  • Brown et al.: Language Models are Few-Shot Learners — Describes GPT-3 as an autoregressive language model pre-trained to predict text, which can then perform tasks from prompts alone; checked October 25, 2026.
  • Google Research: BERT — BERT is pre-trained by masking some input tokens and predicting them from the context on both sides, a different objective from next-token prediction. Checked November 6, 2026.

Go deeper

From the conversation