Models often behave a little differently on data they were trained on than on data they have never seen: they can be more confident, or fit it more tightly. A membership inference attack exploits that gap to work out whether a particular record was part of the training set. Reza Shokri and colleagues showed how to mount it against machine-learning classifiers by training “shadow models” that learn how a target model’s predictions differ between training and non-training inputs, and showed it worked against classifiers trained through commercial machine-learning services from Google and Amazon, including on a hospital discharge dataset (Shokri et al., Membership Inference Attacks Against Machine Learning Models).
NIST’s taxonomy explains why it matters even though it reveals only one bit per record: in some settings, being in the training set is itself sensitive, as with a medical study of patients with a rare disease (NIST AI 100-2e2025, Adversarial Machine Learning). NIST also notes that most of these attacks need only query access, that one of the most efficient and widely used versions just checks whether the model’s loss on a record falls below a threshold, and that membership inference can be a building block for data extraction attacks.
Language models make the extraction side concrete. Nicholas Carlini and coauthors extracted hundreds of verbatim sequences from GPT-2’s training data by querying the model, including names, phone numbers, email addresses and code, even when a sequence appeared in only one training document, and found larger models more vulnerable than smaller ones (Carlini et al., Extracting Training Data from Large Language Models).
The defenses aim at the gap the attack exploits. Shokri’s team found that leakage tracks overfitting and evaluated mitigations such as returning only the top classes, lowering the precision of the probabilities a model returns, and regularization during training. NIST describes differential privacy, which bounds what an attacker can learn about whether a record was included, as the mitigation with a formal guarantee, and keeping sensitive records out of training removes the target entirely. Model inversion covers the related attack that reconstructs data rather than testing for it, and the AI security topic collects the relevant episodes.
Sources
- Shokri et al.: Membership Inference Attacks Against Machine Learning Models — The 2017 shadow-model attack that infers whether a record was in a model's training set.
- NIST AI 100-2e2025: Adversarial Machine Learning — Explains why membership alone can be sensitive, as with a study of patients with a rare disease.
- Carlini et al.: Extracting Training Data from Large Language Models — Extracts memorized training sequences from GPT-2, including ones seen in a single document; larger models were more vulnerable.