A model behind an API is valuable: it cost money to train and may encode private data. An attacker who can query it can send thousands or millions of carefully chosen inputs, record the outputs, and use those pairs to train a look-alike. Tramèr and colleagues showed in 2016 that models offered through public prediction APIs could be extracted this way. With large language models the same idea overlaps with knowledge distillation: training a smaller “student” on a bigger “teacher’s” outputs is a legitimate technique when you own the teacher, and an attack when you don’t have permission.
That overlap is now a policy issue. In episode 68 at 34:35, Chainguard CEO Dan Lorenc lists “distillation attacks” alongside open models and geopolitical incentives among the issues likely to keep policymakers busy. That is his forecast; the episode’s Svix sponsor read names the season’s presenting sponsor. OWASP’s 2025 list for LLM applications includes model extraction via API as a consequence of uncontrolled inference.
Defenses are mostly operational: rate limits and quotas, monitoring for unusual query patterns, limiting how much detail outputs reveal (such as full probability scores), terms of service, and in some cases watermarking outputs. Editorially, extraction sits in the same threat family as prompt injection and data-leak attacks, and belongs in red teaming plans for any model exposed to outsiders.
Hear it from the guest
“Distillation attacks and that kind of thing”
Quotes lightly edited to remove filler words.
Sources
- Tramèr et al.: Stealing Machine Learning Models via Prediction APIs — 2016 paper showing that confidential models deployed behind public query interfaces can be extracted from their predictions; checked October 24, 2026.
- OWASP: LLM10:2025 Unbounded Consumption — Lists model theft and model extraction via API, using crafted queries to collect enough outputs to replicate a model, among the risks of uncontrolled inference; checked October 24, 2026.
- NIST AI 100-2 E2025: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations — NIST taxonomy that covers model extraction among privacy attacks on machine learning systems; checked October 24, 2026.
- MITRE ATLAS — Knowledge base of adversary tactics and techniques against AI systems; checked October 24, 2026.
