Distillation takes a large, capable “teacher” model and uses its outputs to train a much smaller “student.” Instead of learning only from raw labels, the student learns from the teacher’s richer signal — its probability distributions over answers, which Hinton, Vinyals and Dean called “soft targets” in the 2015 paper that named the technique — and ends up punching above the weight its size would suggest. The result is a model that keeps a lot of the teacher’s quality at a fraction of the cost and latency.
It’s one of the main techniques behind capable small models, alongside quantization and good fine-tuning. The limits: the student rarely matches the teacher exactly, it inherits the teacher’s biases and blind spots, and distilling someone else’s model can run into licensing and terms-of-use questions. Used well, it’s how a frontier-grade capability gets cheap enough to run at scale.
Hear it from the guest
“You can try to squeeze as much knowledge, for example, with knowledge distillation into one b model. It will never be as smart as a three b model. That's a fact.”
Quotes lightly edited to remove filler words.
Sources
- Hinton, Vinyals and Dean: Distilling the Knowledge in a Neural Network — Trains a small model on a cumbersome model's class probabilities as soft targets, and reports distilling an ensemble into a single model that is much easier to deploy; checked October 25, 2026.
Go deeper
From the conversation
-
Beyond Transformers: How Liquid AI Is Rethinking LLM Architecture | Maxime Labonne -
How DeepSeek Changed the AI Race Overnight -
AI Codes: Product Engineers Decide What to Build | Laurie Voss, Arize -
Time to Exploit is Negative: AI Broke the Patch Cycle | Dan Lorenc, Chainguard -
Parenting Your AI Agents for Best Results | Cisco's Jeetu Patel -
Every AI Agent Has an Evaluation Gap | Alex Ratner, Snorkel AI -
The Accidental Algorithm | Humans of AI Crossover with Writer's Melisa Russak