Most AI security attention goes to the prompt, but a model is only as trustworthy as the data it learned from. Data poisoning attacks that layer: an adversary slips corrupted or malicious examples into training, fine-tuning, or retrieval data and shapes the model’s behavior before it serves a request. That timing separates it from an evasion attack, which alters inputs at inference time. OWASP lists it for LLM applications as LLM04:2025, Data and Model Poisoning, covering pre-training, fine-tuning and embedding data, and classes it as an integrity attack (OWASP, LLM04:2025 Data and Model Poisoning). NIST’s taxonomy of adversarial machine learning notes that data poisoning attacks apply to every learning paradigm (NIST AI 100-2e2025, Adversarial Machine Learning).
Web-scale training data makes it practical. Nicholas Carlini and coauthors showed two attacks on datasets built from the open internet (Poisoning Web-Scale Training Datasets is Practical). Split-view poisoning exploits the fact that a dataset is a list of URLs: content can change between the time the dataset is assembled and the time someone downloads it, for example after the domain behind a link expires and is bought. They estimate they could have poisoned 0.01% of the LAION-400M or COYO-700M image datasets for about $60. Frontrunning poisoning targets datasets built from periodic snapshots of crowd-sourced sites such as Wikipedia, where an attacker only needs a malicious edit to be live at snapshot time.
The damage is quiet. A poisoned model can pass ordinary evaluation while carrying a bias or a hidden trigger; OWASP notes that poisoning can implant a backdoor that leaves behavior unchanged until a trigger appears. For most teams the exposure is less in pre-training than in what they control: fine-tuning sets, feedback loops, and the documents a RAG system retrieves from, where a planted document works like context poisoning. OWASP’s mitigations start with provenance: track where data came from and how it was transformed, and vet data vendors. Restricting who can write to training and retrieval sources closes the gaps OWASP lists as vulnerabilities. Against split-view poisoning, Carlini’s team proposed distributing cryptographic hashes of all content so later downloads can be checked against what the maintainers indexed; six of the ten affected datasets adopted it. What is context poisoning? covers the retrieval-time version.
Sources
- OWASP: LLM04:2025 Data and Model Poisoning — Covers poisoning of pre-training, fine-tuning and embedding data as an integrity attack on LLM applications.
- NIST AI 100-2e2025: Adversarial Machine Learning — NIST's taxonomy; notes that data poisoning applies to every learning paradigm.
- Carlini et al.: Poisoning Web-Scale Training Datasets is Practical — Two practical attacks on datasets built from the open internet.