Evasion attacks target a model at inference time rather than corrupting its training. The attacker takes an input the model should flag (spam, fraud, malware, disallowed content) and perturbs it just enough that the model misclassifies it while a person still sees what it is. NIST’s taxonomy defines the goal as generating adversarial examples, samples whose classification changes to the attacker’s choice, often with only a minimal perturbation (NIST AI 100-2e2025, Adversarial Machine Learning). NIST dates early instances to 1988, and to 2004 work showing adversarial examples against linear spam filters.
The modern version starts with a 2013 paper by Christian Szegedy and colleagues, who found they could make a deep image classifier misclassify an image by adding a hardly perceptible perturbation, and that the same perturbation often fooled a different network trained on a different subset of the data (Szegedy et al., Intriguing properties of neural networks). Ian Goodfellow, Jonathon Shlens and Szegedy then argued that the main cause is the largely linear behavior of neural networks rather than overfitting, and used that insight for a fast way to generate adversarial examples, which they then used for adversarial training (Goodfellow et al., Explaining and Harnessing Adversarial Examples).
Attackers do not need the model’s weights. NIST notes that adversarial examples work in black-box settings where the attacker can only query the model, including by crafting them on a different model and transferring them, and that the attack has spread from image classification and cybersecurity to speech recognition, natural language processing and video. NIST’s taxonomy covers the generative-AI counterparts, such as jailbreaking, as attacks on language models.
No defense has made models immune. NIST names adversarial training, certified techniques such as randomized smoothing, and formal verification as the most promising directions, notes that each has costs (adversarial training and randomized smoothing can trade accuracy on clean data for robustness), and warns that many published defenses were later broken by stronger attacks. The practical lesson for builders is not to let a single classifier be the only control on a high-stakes decision. Combine model-based detection with rules, rate limits, human review, and guardrails outside the model. What are AI agent guardrails? covers that layering.
Sources
- NIST AI 100-2e2025: Adversarial Machine Learning — Defines evasion as generating adversarial examples whose classification changes, often with a minimal perturbation.
- Szegedy et al.: Intriguing properties of neural networks — Hardly perceptible perturbations that cause misclassification and often transfer to other networks.
- Goodfellow et al.: Explaining and Harnessing Adversarial Examples — Attributes adversarial examples to linear behavior and introduces a fast generation method used for adversarial training.