A trained model encodes information about its training data, and a model inversion attack tries to pull that information back out by querying the model and analyzing its responses. Matt Fredrikson, Somesh Jha and Thomas Ristenpart showed in 2015 that the confidence scores many machine-learning services return alongside predictions are enough to mount the attack (Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures). Against a decision tree trained on a lifestyle survey, they could estimate whether a respondent had admitted to cheating on a partner. Against a facial recognition network, they recovered recognizable images of people’s faces given only a name and query access to the model.
NIST’s taxonomy files model inversion under privacy attacks, next to data reconstruction, membership inference and model extraction (NIST AI 100-2e2025, Adversarial Machine Learning). It draws a useful distinction: classic model inversion produces representative examples of a class that resemble the training data rather than exact training records, while later reconstruction attacks have recovered actual training samples under stronger assumptions about the attacker’s access. NIST connects both to the tendency of neural networks to memorize training data.
The risk applies wherever a model trained on sensitive data answers outside queries: medical, financial, biometric, or employee data. Fredrikson’s paper also tested countermeasures, including returning only rounded confidence values, and concluded the attacks could be avoided with negligible loss of utility. The general lesson is to return less: rounded or withheld confidence scores, and labels instead of full probabilities. NIST describes differential privacy, which bounds how much anyone can learn about an individual record from a model’s output, as a mitigation with a formal guarantee against reconstruction and membership attacks, and keeping sensitive data out of training limits what any attack can recover.
Model inversion sits alongside membership inference, which asks only whether a record was in the training set. For LLM applications, OWASP covers this exposure under sensitive information disclosure (LLM02:2025), noting that revealed training data can enable inversion attacks. The AI security topic collects the episodes on attacks against models, and why enterprises need on-prem AI covers keeping sensitive data under your own control.
Sources
- Fredrikson, Jha and Ristenpart: Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures — The 2015 paper using returned confidence scores to reconstruct recognizable faces from a name and query access.
- NIST AI 100-2e2025: Adversarial Machine Learning — Files model inversion under privacy attacks beside data reconstruction, membership inference and model extraction.