Model Distillation: How Smaller AI Models Learn From Bigger Ones
Model distillation trains a small, fast AI model to mimic a larger one, so products can run cheaper and faster without starting from scratch.
In this story 6 sections
Model distillation is a training technique where a smaller "student" model learns to reproduce the outputs of a larger "teacher" model, capturing much of its capability at a fraction of the size and cost.
Running the biggest available AI model on every single request is slow and expensive, and most tasks don’t actually need that much horsepower. Distillation is how labs get a smaller model that performs close to a bigger one without training it from raw data all over again.
This guide covers how model distillation actually works, why it has become so central to the industry’s economics, and what a distilled model gives up compared to the original. It’s written for anyone deciding between a flagship model and a smaller distilled version for a real product.
The technique isn’t new — it predates the current AI boom by nearly a decade — but its role has shifted from a research curiosity to a core part of how every major lab ships products at different price points.
How Distillation Actually Trains a Smaller Model
In a typical distillation setup, the smaller student model is trained not just on the correct answer to a question, but on the full probability distribution the teacher model assigned across possible answers. That distribution carries more information than a single right answer, since it tells the student how confident the teacher was and what alternatives it considered.
This is why a distilled model often generalizes better than one trained from scratch on the same amount of raw data — it’s learning from the teacher’s reasoning pattern, not just its final output. Researchers sometimes call this "soft label" training, as opposed to the "hard label" training used for a model built from raw data alone.
The teacher model doesn’t need to be involved at inference time at all once training finishes. The whole point is that the student model, once trained, runs entirely on its own — smaller, faster, and independent of the teacher it learned from.
Why Labs Distill Models Instead of Just Building Small Ones
Training a good small model from scratch is genuinely hard — smaller models have less capacity to learn nuance directly from raw internet-scale data. Distillation sidesteps that by letting the small model learn from a teacher that already extracted the useful patterns.
It’s also a way to make a family of models with consistent behavior at different price points. A company can offer a flagship, a mid-size, and a lightweight model that all "feel" similar in tone and judgment because the smaller ones were distilled from the same teacher, rather than trained independently with their own quirks.
This matters economically too. We’ve seen at Emergent Wire that inference cost, not training cost, dominates a model’s long-run expense once it ships to millions of users, so a distilled model that’s 80% as capable at 10% of the compute cost is often the better business decision.
Independent research from Epoch AI, a nonprofit that tracks AI compute trends, has found that inference costs for widely used models have fallen sharply year over year, with smaller distilled and compressed models accounting for a growing share of that decline.
What a Distilled Model Actually Gives Up
The biggest loss is usually in multi-step reasoning and edge cases — tasks that require holding several pieces of context in mind at once tend to degrade faster than simple factual recall. A distilled model can often still answer "what’s the capital of France" perfectly while struggling with a five-step logic puzzle the teacher handled fine.
Distilled models also tend to be less robust to unusual phrasing or adversarial prompts, since the teacher’s broader training gave it more exposure to edge cases the smaller model never directly learned. This isn’t a hard rule — a well-distilled small model can outperform a poorly trained larger one — but it’s the general pattern worth expecting.
For anyone comparing a flagship model against its distilled sibling, the gap is usually smallest on straightforward tasks and largest on genuinely novel, multi-step ones — worth testing directly on your actual use case rather than trusting a general benchmark comparison alone.
Distillation vs. Quantization vs. Pruning
These three techniques are often combined rather than used alone. A model might first get distilled down to a smaller architecture, then quantized further for deployment on a phone or edge device, stacking the savings from each step.
The National Institute of Standards and Technology (NIST) has published benchmarking guidance specifically noting that compressed models — whether via distillation, quantization, or pruning — need separate evaluation from their full-size counterparts, since standard benchmarks can overstate real-world reliability after compression. The compute savings from this stacking are part of why the industry’s power demands, covered in our report on the AI compute buildout, haven’t grown as fast as raw model capability has.
| Technique | What It Does | Typical Size Reduction |
|---|---|---|
| Distillation | Trains a new, smaller model to mimic a larger one | 5-20x |
| Quantization | Reduces numeric precision of existing weights | 2-4x |
| Pruning | Removes redundant connections from an existing model | 1.5-3x |
When a Distilled Model Is the Right Choice
If your product involves simple, high-volume tasks — classification, short answers, basic summarization — a distilled model is usually the better economic choice, since the accuracy gap rarely matters for the task at hand. Save the flagship model’s extra capability for genuinely hard, low-volume requests.
This overlaps heavily with the tradeoffs we covered in our piece on small language models running on-device, since many on-device models are themselves distilled versions of larger cloud models, shrunk further to fit local hardware constraints.
For anyone building a system that routes between model sizes automatically, it’s worth reading our piece on mixture of experts, a related but distinct approach to getting more efficiency out of a single model rather than a family of separate ones. Many production systems combine both ideas today.
The right test is always the same: run your actual worst-case examples through both the flagship and the distilled version, and see how often the smaller model’s answer would have actually mattered to a real user. Often, it doesn’t.
The Bottom Line
Model distillation is one of the quieter but more consequential techniques in AI right now, because it’s what makes AI assistants affordable enough to run at massive scale. Most of what you interact with day to day, even inside a "flagship" branded app, is probably a distilled or otherwise compressed model handling the routine load.
Understanding the tradeoff — faster and cheaper, at some cost to hard reasoning — helps explain why the same company’s different-tier products can feel meaningfully different even when marketed similarly.
Emergent Wire covers AI models, capabilities, and the industry building them for readers who want the real story behind the demos.
