Continued pre-training uses more unlabeled text to update an existing model. Supervised fine-tuning instead uses labeled input-output pairs; AWS documents the distinction for Nova (AWS, Continued pre-training). Gururangan and colleagues studied domain-adaptive and task-adaptive pre-training across four domains and found gains on their classification tasks (Don’t Stop Pretraining). Those results support testing domain adaptation, not assuming every corpus will help.
For example, a legal-search team might train on a collection of legal text, then separately tune on examples of the answers it wants. The raw documents supply domain language and patterns; the demonstrations specify response behavior. Compare the adapted checkpoint with the starting model on held-out legal questions and the general tasks your product still needs.
Describing one stage of Thomson 1’s pipeline, Joel Hron says, “The next thing that we do in terms of model training involves continuous pre-training.” (episode 69, 14:37).
Choose a checkpoint and training-data mixture against that acceptance test. AWS’s checkpoint and data-mixing guidance discusses preserving existing skills while adding domain data. Catastrophic forgetting names the regression risk. Fine-tuning covers task adaptation, and fine-tuning vs RAG vs prompting helps decide whether changing weights is the right next experiment.
Sources
- AWS, Continued pre-training — Primary source for the explanation and distinctions in this entry.
- Don't Stop Pretraining — Primary source for the explanation and distinctions in this entry.
- checkpoint and data-mixing guidance — Primary source for the explanation and distinctions in this entry.
