Knowledge Distillation

Distillation takes a large, capable “teacher” model and uses its outputs to train a much smaller “student.” Instead of learning only from raw labels, the student learns from the teacher’s richer signal — its probability distributions over answers — and ends up punching above the weight its size would suggest. The result is a model that keeps a lot of the teacher’s quality at a fraction of the cost and latency.

It’s one of the main techniques behind capable small models, alongside quantization and good fine-tuning. The limits: the student rarely matches the teacher exactly, it inherits the teacher’s biases and blind spots, and distilling someone else’s model can run into licensing and terms-of-use questions. Used well, it’s how a frontier-grade capability gets cheap enough to run at scale.