Catastrophic Forgetting
Catastrophic forgetting is a substantial loss of previously learned capabilities when a neural network trains on new tasks or data. A model can improve in a specialty while getting worse at skills it had already learned.
Also known as: catastrophic interference
Model TrainingAI Evaluation & Reliability
Training changes the model’s weights. Updates that help with new material can interfere with weights that support earlier tasks, so a legal model might gain domain knowledge while losing coding ability. Researchers have shown that constraining changes to weights important for earlier tasks can reduce this damage. Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks.
Measure both sides of the tradeoff: performance on the new domain and the capabilities the model needs to retain. Mixing general training data with domain data is another way to limit forgetting during continued pre-training, but the mixture needs evaluation rather than an assumption that more specialization is always better. AWS’s guidance on data mixing describes that process. Joel Hron discusses checking general capabilities alongside legal performance in the Thomson 1 episode below.
