Continued Pre-training
Continued pre-training extends a pretrained model's training on additional unlabeled data, often to adapt it to a domain or language. It updates the model's weights using a pre-training objective.
Also known as: continuous pre-training, continual pre-training
A team can start with an existing language model and continue training it on legal documents, technical manuals, or another corpus. The training examples come from the text itself. Supervised fine-tuning uses labeled examples, such as instructions paired with desired responses, to teach particular behavior. AWS’s continued pre-training documentation explains this distinction.
Continued pre-training often precedes instruction tuning, but it can also start from a model that has already been instruction-tuned. The choice depends on the available checkpoint and the adaptation goal. Further training can cause catastrophic forgetting, so evaluate retained capabilities as well as gains in the new domain. AWS’s checkpoint guidance covers these choices. Joel Hron describes continued pre-training as one stage in Thomson 1’s model-development pipeline in the episode below.
