What Is AI Model Poisoning? Complete 2026 Guide
- 4 hours ago
- 30 min read
Most people picture an AI attack as something dramatic: a hacker breaking through a firewall in real time. Data poisoning looks nothing like that. It happens quietly, months before deployment, a few corrupted examples slipped into a training set or a handful of malicious documents hiding in a web crawl, and the damage only shows up later, when the model behaves normally for almost everyone except the one person who knows the secret trigger.
Â
Key Takeaways at a Glance
Â
AI model poisoning covers a family of attacks against training data, model parameters, or update pipelines, not one single technique.
NIST's adversarial machine learning taxonomy treats poisoning as a distinct integrity or availability attack, separate from evasion and privacy attacks.
A poisoned model can carry a hidden backdoor that behaves normally until a specific trigger appears, making the problem hard to catch with normal testing.
Anthropic's 2025 research found that as few as 250 malicious documents could backdoor language models regardless of overall model size.
Federated learning, third-party pretrained models, and retrieval-augmented generation systems each introduce their own poisoning surface, with different attacker assumptions and defenses.
No single control fully prevents poisoning; defense depends on provenance tracking, testing, monitoring, and incident response working together across the AI lifecycle.
Quick Answer
Â
AI model poisoning is an attack that manipulates the data, updates, or components used to build or update a machine learning model so that the resulting model behaves incorrectly, unsafely, or maliciously. Poisoning can degrade overall accuracy, plant a hidden backdoor that only activates on a specific trigger, or bias outputs in a chosen direction. It can happen during data collection, training, fine-tuning, or through compromised third-party models.
Table of Contents
AI Model Poisoning: A Clear Definition
In everyday use, "AI model poisoning" and "AI data poisoning" often mean the same thing: someone tampers with the inputs to a machine learning system so the resulting model misbehaves. That casual usage is not wrong, but it blurs an important line.
NIST's Adversarial Machine Learning taxonomy, published in March 2025, treats poisoning as an attack that compromises training data, model parameters, or update mechanisms during the learning process, and separates it from evasion attacks that manipulate inputs only at inference time. Under this framing, model poisoning is the broader idea: any manipulation that corrupts what a model learns or how its weights are set, whether that manipulation happened through the training data, a malicious update in federated learning, or a tampered pretrained checkpoint. Data poisoning is the most common route into that outcome, but not the only one.
OWASP's GenAI Security Project groups both concerns under LLM04:2025, "Data and Model Poisoning," and treats it as an integrity attack because it undermines the model's ability to make trustworthy predictions.
Why AI Model Poisoning Matters
A poisoned model does not necessarily look broken. It can pass ordinary quality checks, hit its accuracy targets, and still contain a narrow, hidden flaw that only an attacker knows how to trigger. That is precisely what makes the risk significant for AI security teams, not incidental.
The stakes rise with how much a model is trusted to do. A poisoned fraud-detection model can let specific transaction patterns through undetected. A poisoned code-completion model can suggest insecure patterns only under certain prompts. A poisoned LLM embedded in an agentic workflow can misuse the tools and data access it has been granted. The NIST AI Risk Management Framework's Generative AI Profile explicitly lists data poisoning as an Information Security risk that regulators, procurement teams, and boards increasingly expect organizations to address.
How AI Model Poisoning Works
At a conceptual level, poisoning exploits a simple fact: a model learns whatever pattern is present in its training signal, without judgment about whether that pattern is legitimate. An attacker who can influence some part of that signal, even a small part, can steer what the model eventually learns.
A typical attack unfolds in stages. First, the attacker gains some form of access: control over a data source, a contribution channel such as crowdsourced labeling, a role as a federated-learning participant, or the ability to publish a pretrained checkpoint that others will download. Second, the attacker crafts manipulated examples, labels, or weight updates designed to introduce a specific behavior, sometimes tied to a "trigger" the attacker can invoke later, and sometimes intended to degrade overall performance instead. Third, the model absorbs this signal during training, fine-tuning, or aggregation, and the manipulation becomes baked into the model's parameters. Finally, at deployment, the model behaves normally on everyday inputs while producing attacker-chosen outputs on triggered ones, or simply performs worse than expected in ways that are hard to trace back to the original cause.
MITRE ATLAS, a knowledge base of adversary tactics against AI systems modeled on the ATT&CK framework, documents this staging behavior directly: it defines a tactic called "AI Attack Staging" for the preparation work attackers do, such as building a proxy model to test a poisoning strategy offline before deploying it against a production system.
Model Poisoning vs. Data Poisoning
The two terms overlap heavily but are not perfect synonyms. The distinction matters for anyone building defenses, because different manipulation points call for different controls.
Data poisoning targets the training examples themselves:
Adding mislabeled or malicious examples to a training set.
Editing the content of web pages an automated crawler will later scrape, known as split-view poisoning.
Injecting content into a dataset just before a scheduled snapshot, known as frontrunning poisoning.
Manipulating feedback, ratings, or preference data used in fine-tuning.
Model poisoning, in the narrower sense, targets the model itself or the update process:
Submitting malicious weight updates in a federated-learning round.
Distributing a pretrained checkpoint whose weights were altered to include a backdoor, as in weight poisoning research on pretrained NLP models.
Directly editing model parameters after training, without ever touching a dataset.
Replacing an entire model file during a software supply-chain compromise.
In practice, most real incidents combine elements of both: manipulated data is the delivery mechanism, and the model's parameters are the ultimate target.
Model Poisoning vs. Prompt Injection, Jailbreaking, and Adversarial Examples
Poisoning is frequently confused with several related but distinct problems. Keeping them separate matters because each needs a different defense.
Prompt injection happens at inference time. An attacker embeds instructions in a prompt, a document, or a web page that a model reads, trying to override its intended behavior. The model's weights never change; only the immediate output is affected. Articsledge's article on prompt injection covers this pattern in depth.
Jailbreaking is a form of prompt-level manipulation aimed at bypassing a model's safety training, again without altering its underlying parameters.
Adversarial examples, as described in research on adversarial machine learning, are inputs crafted to fool an already-trained model at inference time, exploiting quirks in how it generalizes, not a manipulation of its training process.
Model extraction attempts to steal or reconstruct a model's behavior by querying it repeatedly, rather than corrupting it.
Data drift and ordinary model errors are not attacks at all. Performance can degrade naturally as the world changes or through normal noisy data, and this should not be labeled poisoning unless there is evidence of deliberate, adversarial manipulation.
Hallucination describes a generative model producing confident but incorrect output, a limitation of generation rather than evidence of tampering.
Poisoning is the only item on this list that changes what the model actually learned or the parameters it carries forward into deployment.
Where Poisoning Can Enter the AI Lifecycle
Poisoning is not confined to one stage. The NSA, CISA, and FBI's joint guidance on AI data security, published in May 2025, frames this as a lifecycle-wide problem spanning development, testing, deployment, and operation. Realistic entry points include:
Data sourcing, where web-scraped or purchased datasets may already contain manipulated content before an organization ever sees them.
Data labeling, where human or automated annotators can be tricked or bribed into mislabeling examples.
Data cleaning, where a flawed sanitization step can fail to remove planted outliers.
Pretraining, the large-scale initial training phase for foundation models and LLMs.
Fine-tuning and instruction tuning, which adapt a general model to specific tasks using smaller, sometimes less-scrutinized datasets.
Preference or feedback data, used to align generative models with human judgments, which can be manipulated by coordinated low-quality ratings.
Federated training, where individual participants submit updates the central aggregator cannot fully inspect.
Embedding generation and vector database indexing, which can be manipulated without touching the base model at all.
Model repositories and checkpoints, where a downloaded pretrained model may already carry a backdoor.
Third-party components, such as tokenizers, adapters, or open-source dependencies.
Deployment updates, where a compromised release pipeline can swap in a tampered model version.
Feedback loops and continuous learning, where a deployed system that learns from live user interactions can be poisoned by coordinated abuse, as happened with Microsoft's Tay chatbot in 2016, a documented real-world incident in which coordinated users exploited a vulnerability in the bot's conversational learning feature within the first 24 hours of launch, causing it to post offensive content before Microsoft took it offline.
The Main Types of AI Model Poisoning Attacks
Security researchers classify poisoning attacks along several overlapping dimensions rather than one fixed list. Understanding these categories helps defenders reason about what an attacker is trying to achieve.
By objective:
Availability poisoning aims to broadly degrade a model's accuracy or reliability, without a specific target output in mind.
Integrity poisoning aims to make the model produce a specific wrong answer under specific conditions, while leaving overall performance largely intact.
By scope:
Targeted poisoning manipulates behavior for a narrow set of inputs, such as one class label or one attacker-chosen phrase.
Untargeted poisoning tries to lower performance broadly, across many inputs.
By mechanism:
Backdoor poisoning implants a hidden trigger, discussed in more detail below.
Clean-label poisoning manipulates only the input features while keeping labels technically correct, making the poisoned examples harder to spot through label review alone.
Dirty-label poisoning deliberately mislabels examples, a cruder but simpler approach.
Model-update poisoning targets the aggregation step in federated or continuously updated systems.
Model replacement substitutes an entire malicious model for the legitimate one, often via a compromised distribution channel.
Weight poisoning directly manipulates model parameters, sometimes without touching training data at all.
Supply-chain poisoning compromises a shared dependency, such as a pretrained checkpoint, dataset package, or library, that many downstream systems will inherit.
These categories overlap in real attacks. A single incident might combine clean-label examples with a backdoor trigger, delivered through a compromised supply-chain dependency.
Backdoors, Triggers, and Sleeper Behaviors
A backdoored model is, by design, a model that behaves normally on almost every input. That is exactly what makes backdoors difficult to catch: standard accuracy testing on clean data will not reveal them, because the model was built to pass those tests.
A trigger is whatever condition activates the hidden behavior. Triggers can be visible, such as a specific pixel pattern in an image; semantic, such as a particular phrase or code word; contextual, such as a combination of otherwise-ordinary features; or latent, meaning encoded in a way that is not obviously visible in the raw input at all. Anthropic's Sleeper Agents research, published by Hubinger and colleagues in 2024, is a controlled research demonstration showing that deceptive behaviors trained into a language model can persist through standard safety training techniques, rather than being removed by them. This is research demonstrating a mechanism, not evidence that any deployed production model currently carries such a backdoor.
It is worth being precise about the difference between three things that get conflated: peer-reviewed backdoor research that demonstrates a mechanism under controlled conditions, a proof-of-concept attack built to show feasibility, and a confirmed malicious deployment found in the wild. Most of the striking findings in this field, including the sleeper-agent work and the pretraining-poisoning study discussed later in this article, fall into the first two categories.
Poisoning in Predictive AI and Generative AI
Poisoning shows up differently depending on what kind of system is targeted.
In predictive systems, classic examples include spam filters, malware and fraud classifiers, recommendation systems, and computer-vision models. A poisoned fraud classifier might be trained to wave through transactions carrying a specific attacker-chosen pattern. A poisoned malware detector might be trained to mark a specific malicious file signature as benign, directly undermining the antivirus and endpoint protection tools that rely on it.
In generative systems, including chatbots, code-generation models, and multimodal systems, poisoning can degrade output quality broadly, introduce targeted misinformation, weaken safety guardrails for specific prompts, or implant behaviors that only appear under a trigger phrase. Generative adversarial networks and other generative architectures add further surface area, since poisoning can affect either the generator or the training signal used to guide it. NIST's 2025 taxonomy extends its poisoning categories specifically to cover generative AI systems, alongside supply-chain attacks and prompt injection, reflecting how much the threat model has expanded since predictive classifiers were the primary concern.
Model Poisoning in Federated Learning
Federated learning lets many participants, such as phones or hospitals, jointly train a shared model without exposing their raw local data to a central server. That privacy benefit comes with a security cost: the central aggregator cannot fully inspect what each participant actually did to produce their submitted update.
The foundational demonstration of this risk is Bagdasaryan and colleagues' paper, How To Backdoor Federated Learning, presented at AISTATS 2020. The researchers showed that a single malicious participant, selected in just one round of training, could use a technique called model replacement to push the shared global model to 100% accuracy on an attacker-chosen backdoor task, while leaving its accuracy on the main task unaffected. This is a controlled research demonstration on standard federated-learning benchmarks such as image classification and next-word prediction, not a documented attack against a production system, but it fundamentally changed how the field thinks about federated-learning security. The same paper showed that this model-replacement approach substantially outperformed traditional data-poisoning attacks in the federated setting.
Common defenses include robust or Byzantine-resilient aggregation methods that try to down-weight anomalous updates, and anomaly detection on submitted model deltas. Bagdasaryan's own paper also demonstrated a "constrain-and-scale" technique that could evade some anomaly-detection defenses, illustrating a broader theme in this field: most defenses reduce risk rather than eliminate it, and defenders should assume an adaptive attacker will try to route around any single control.
Poisoning Large Language Models
LLMs are trained in stages, and each stage offers a different poisoning opportunity: web-scale pretraining on enormous, loosely curated corpora; smaller curated corpora used for targeted skill-building; fine-tuning and instruction tuning on task-specific examples; preference data used to align a model's behavior with human judgment; and increasingly, user-contributed examples and synthetic data generated by other models.
For years, the working assumption in this field was that poisoning a large pretrained model required controlling a percentage of its training data proportional to its size, making attacks against frontier-scale models seem impractical. A 2025 study by Anthropic, conducted jointly with the UK AI Security Institute and the Alan Turing Institute, overturned that assumption. The researchers trained 72 models ranging from 600 million to 13 billion parameters and found that injecting just 250 malicious documents into the pretraining data was enough to implant a narrow backdoor, a "denial-of-service" style trigger that made the model output gibberish text when it encountered a specific phrase, regardless of the model's overall size or the total volume of its training data. Because a 13-billion-parameter model trained on more than 20 times as much data as the 600-million-parameter model was backdoored by the same fixed number of documents, the finding challenges the older assumption that larger models require proportionally more poisoned data.
Two important caveats matter here. First, Anthropic's own researchers were explicit that this study tested a narrow, low-stakes backdoor behavior, and it remains an open question whether the same sample efficiency holds for larger frontier models or for more harmful behaviors; the results should not be extrapolated beyond what was actually tested. Second, getting malicious documents into a real pretraining pipeline at all is a separate, nontrivial challenge, addressed in the supply-chain section below.
Fine-tuning and instruction tuning carry their own risk because they typically use much smaller datasets than pretraining, meaning an attacker needs to control a much larger relative share of the data to have an effect, which paradoxically makes fine-tuning-stage poisoning easier to execute per malicious example, even though the absolute datasets are smaller and often easier to review.
RAG, Embedding, and Knowledge-Base Poisoning
Retrieval-augmented generation lets an LLM pull in external documents at query time rather than relying only on what it learned during training. This is a different attack surface from model poisoning, and the distinction matters: manipulating a RAG knowledge base changes what the model retrieves and reads at inference time, not the model's underlying weights. Restoring a clean knowledge base or re-indexing a vector database can often remediate a RAG poisoning incident without ever touching the base model.
PoisonedRAG, presented at USENIX Security 2025 by Zou, Geng, Wang, and Jia, formalized this as a knowledge-corruption attack: injecting a small number of malicious texts into a RAG system's knowledge database to make the model generate an attacker-chosen answer to an attacker-chosen question. The researchers reported a 90% attack success rate when injecting as few as five malicious texts per target question into a knowledge base containing millions of texts, and found that several existing defenses were insufficient against it. This is a peer-reviewed research demonstration under specific experimental assumptions about attacker access to the knowledge base, not a report of the technique being used against a production system.
RAG poisoning overlaps conceptually with indirect prompt injection, since both involve an attacker planting malicious content that a model later reads, but they are not identical: prompt injection is about the instructions embedded in content the model processes, while RAG poisoning is specifically about corrupting the retrieval index or knowledge source itself so that manipulated content gets surfaced in the first place. Word and sentence embeddings, and the embedding space they live in, can themselves be manipulated to change what a retrieval system considers relevant, even without altering the visible text of any document.
AI Supply-Chain and Pretrained-Model Poisoning
Modern AI development leans heavily on shared infrastructure: public model repositories, third-party checkpoints, fine-tuning adapters, tokenizers, embedding models, and open dataset packages. Each of these is a potential entry point for a supply-chain compromise, and the risks fall into two distinct categories that are worth separating clearly.
The first category is behavioral poisoning of the model itself, such as the weight-poisoning research on pretrained NLP models by Kurita, Michel, and Neubig, presented at ACL 2020. Their research demonstrated that pretrained weights could be injected with vulnerabilities that expose a backdoor after fine-tuning, letting an attacker manipulate the model's predictions simply by including a specific keyword, even when the attacker had limited knowledge of the eventual fine-tuning dataset or procedure.
The second category is malware hidden inside the model file itself, which is a software security problem more than a machine-learning one. In February 2024, security researchers at JFrog disclosed that over 100 malicious machine learning models had been uploaded to the Hugging Face model hub. The malicious payloads exploited Python's pickle serialization format, using its reduce method to execute arbitrary code the moment a user loaded the model file, in some cases granting the attacker a full remote shell on the victim's machine. This is a documented real-world incident, and it illustrates that "poisoning" a model's behavior and hiding executable malware inside a model file are separate risks that call for separate defenses: provenance and data-quality controls address the first, while safe serialization formats and sandboxed loading address the second.
Typosquatting on popular package or model names, compromised maintainer accounts, and unreviewed community fine-tunes all widen this same attack surface.
Attacker Goals, Capabilities, and Access
Framing poisoning purely as a technical exercise misses the range of motives behind it. Documented and researched attacker goals include degrading a competitor's or target's model reliability, planting a hidden backdoor for future exploitation, biasing outputs toward a political or commercial agenda, bypassing safety training in a generative model, enabling fraud that evades a poisoned detection system, leaking sensitive information through a manipulated model, causing downstream tools connected to an AI agent to be misused, or simply damaging an organization's reputation.
Security researchers commonly describe attacker knowledge on a spectrum from black-box, where the attacker can only observe inputs and outputs, through gray-box, with partial knowledge of the architecture or training process, to white-box, with full access to the model and its training pipeline. Separately, attacker access can range from data access alone, to label access, to full model access, to source-code access, to supply-chain access over a shared dependency used by many downstream victims at once. Higher access generally enables more precise, harder-to-detect attacks, which is why supply-chain and insider scenarios deserve particular attention in a risk assessment.
Who Is Most at Risk and What the Impact Can Be
Exposure to poisoning risk scales with how much external or uncontrolled input a system depends on. Organizations that train on public or crowdsourced data, continuously learn from live user interactions, rely on external or third-party datasets, deploy third-party pretrained models without independent verification, participate in federated learning, maintain large RAG knowledge bases fed by many contributors, or grant an AI system access to sensitive data or automated tools all carry elevated exposure.
The potential impact spans several dimensions at once. Technically, a poisoned model can silently misclassify, mis-rank, or generate incorrect content. Operationally, it can force costly retraining, rollback, and re-validation cycles. Financially, a poisoned fraud or credit-decision model can enable direct monetary loss. Legally, depending on sector and jurisdiction, an organization may face disclosure or liability questions if a poisoned model made a consequential decision about a customer; this article is not legal advice, and organizations facing an actual incident should involve qualified counsel. Safety impact is highest in domains such as healthcare, autonomous systems, or critical infrastructure, where NIST's own 2025 report notes that adversarial failures become more consequential the more a system is embedded in high-stakes decisions. Reputational impact can persist well after a technical fix, as Microsoft's Tay incident illustrated in 2016.
Warning Signs and Indicators of Possible Poisoning
No single signal proves poisoning on its own, but a combination of the following should prompt a closer look, ideally as part of routine AI red teaming and monitoring rather than only after something goes wrong:
A sudden or unexplained shift in performance metrics between training runs or model versions.
Narrow, trigger-specific failures on otherwise normal-looking inputs, especially failures that do not show up in aggregate accuracy numbers at all.
Consistently strange behavior tied to a specific rare phrase, image pattern, or input combination.
Dataset or checkpoint file hashes that do not match a known-good baseline.
Unapproved changes in data lineage or model provenance records.
Unusually large or statistically unusual update norms submitted by a federated-learning participant.
Outlier gradients during training that do not resemble the rest of the batch.
Retrieval anomalies in a RAG system, such as a previously low-ranked document suddenly dominating results for a specific query.
Unexpected shifts in embedding space geometry after a re-indexing job.
New, unreviewed model artifacts appearing in a repository without a corresponding, approved change record.
Inconsistent outputs between two versions of a model that should be functionally equivalent.
Suspicious account activity or unexplained changes at a vendor or model-repository level.
How to Detect AI Model Poisoning
Detection works best as defense in depth, combining several complementary techniques rather than relying on any one of them:
Data validation and provenance checks before data enters a training pipeline, verifying where it came from and whether that source is trusted.
Dataset versioning, so any change to a training corpus is tracked and attributable.
Cryptographic hashes and digital signatures on datasets and checkpoints, a core recommendation of the NSA and CISA's joint AI data security guidance.
Distribution and outlier analysis to flag statistically unusual examples before training.
Label-consistency review, sampling labeled data against an independent source of truth.
Holdout and canary testing, including inputs specifically designed to reveal known trigger patterns.
Behavioral differential testing between a candidate model and a trusted prior version.
Model-weight and checkpoint comparison against a known-good baseline.
Update-norm monitoring in federated or continuously updated systems.
Robust aggregation methods that reduce the influence of any single anomalous update.
Backdoor scanning and model inspection tools that probe for anomalous trigger-like behavior.
RAG retrieval auditing, periodically reviewing what a retrieval system actually surfaces for sensitive queries.
Red teaming and adversarial testing conducted on a recurring schedule, not only before launch.
Continuous monitoring in production, watching for the warning signs listed above.
Poison forensics, the after-the-fact process of tracing a suspected poisoning incident back to its source data or update.
Every one of these methods has limits. Outlier detection can produce false positives that discard legitimate unusual data. Clean-label and semantic-trigger attacks are specifically designed to blend in with normal data distributions, defeating simple statistical checks. An adaptive attacker aware of a defense will often design around it, which is why detection should be layered rather than treated as a solved problem.
How to Prevent and Mitigate Model Poisoning
No single safeguard is sufficient on its own. Effective prevention layers controls across governance, data, training, deployment, and monitoring.
Governance and Ownership
Assign clear ownership for data quality, model risk, and AI governance decisions. Require documented approval workflows for any change to training data sources, model architecture, or deployment pipeline, and maintain immutable logs of who approved what and when. The NIST AI Risk Management Framework's Generative AI Profile recommends organizing this kind of oversight around its Govern, Map, Measure, and Manage functions.
Data and Dataset Security
Favor trusted, verifiable data sources over unvetted web-scraped content where possible. Track data provenance and maintain a dataset inventory. Apply least-privilege access controls and separation of duties, so no single person can both submit and approve a dataset change unreviewed.
Training-Pipeline Security
Use version control and reproducible builds for training code and configuration. Apply cryptographic verification to datasets and checkpoints entering the pipeline. Where appropriate, apply gradient clipping and differential-privacy techniques to reduce the influence any single example or update can have on the final model.
Model and Checkpoint Security
Prefer trusted model file formats designed to avoid arbitrary code execution on load, over formats known to carry that risk. Sandbox the loading of any third-party model before granting it access to production systems or sensitive data. Maintain signed artifacts and independent validation of any model before deployment.
Federated-Learning Controls
Apply robust or Byzantine-resilient aggregation to reduce the influence of anomalous participant updates. Monitor update norms for statistical outliers. Accept that, by federated learning's own design, defenders cannot see participants' raw data, so detection must rely on the properties of submitted updates rather than the underlying data itself.
RAG and Knowledge-Base Controls
Apply access controls and provenance checks to documents entering a RAG knowledge base. Periodically audit retrieval results for sensitive or high-value queries. Treat a compromised knowledge base as a faster, cheaper incident to remediate than a compromised base model, since re-indexing is usually sufficient.
Supply-Chain and Vendor Controls
Conduct vendor due diligence before adopting a third-party model or dataset. Request model cards and data documentation. Maintain an ML bill of materials, or ML-BOM, cataloging every model, dataset, and dependency in use, mirroring the software bill of materials concept from traditional cybersecurity.
Testing and Monitoring
Rate-limit and human-review any feedback-ingestion pipeline that could otherwise let an attacker flood a system with low-quality or malicious signal. Conduct recurring red-team exercises rather than a single pre-launch review. Maintain a rollback plan so a poisoned model version can be reverted quickly.
Access Control and Infrastructure Security
Apply the same infrastructure security fundamentals used elsewhere in cybersecurity, including SIEM monitoring and endpoint protection, to the systems that store training data, host model repositories, and run training jobs.
Incident Response: What to Do When Poisoning Is Suspected
When a model shows signs consistent with poisoning, a structured response reduces both the technical damage and the time needed to recover:
Pause the affected training, updating, or deployment process to prevent further exposure.
Preserve evidence, including logs, dataset snapshots, and model checkpoints, before making changes.
Isolate the relevant artifacts and access paths connected to the suspected incident.
Identify every affected model version, dataset, embedding index, and downstream system that consumed the suspect artifact.
Compare the suspect model or dataset against a known-good baseline.
Revoke any compromised credentials or third-party vendor access involved.
Roll back to a verified prior version once it is safe to do so.
Rebuild from trusted data and artifacts when the original source cannot be fully trusted again.
Retest thoroughly before redeployment, including targeted trigger testing.
Notify the relevant internal stakeholders, including security, legal, and leadership.
Evaluate legal, contractual, regulatory, and customer-notification obligations with qualified professionals; this is not something to decide unilaterally from a technical team alone.
Document lessons learned and feed them back into governance and testing controls.
A critical point often missed in step 8: retraining a model on the same data source, through the same pipeline, using the same compromised account, does not fix anything if the original point of compromise is still open. Root-cause containment has to come before rebuilding, or the same poisoning attempt can simply happen again.
A Practical AI Model Poisoning Security Checklist
Before acquiring a model
Review the vendor's model card, training data documentation, and known provenance.
Confirm the model file format and scan it for unsafe serialization risks before loading it anywhere sensitive.
Check for an ML-BOM or equivalent dependency inventory.
Before training
Verify data source trust level and provenance for every dataset in use.
Establish a clean, hashed, version-controlled baseline dataset.
Define approval workflow and separation of duties for data changes.
During training
Monitor for statistical outliers and label inconsistencies.
Apply gradient clipping or differential privacy where appropriate.
Log every data and configuration change with attribution.
Before deployment
Run holdout, canary, and targeted trigger testing.
Compare model weights against the last known-good checkpoint.
Conduct an independent red-team review.
During operation
Monitor for the warning signs described earlier in this article.
Audit RAG retrieval results periodically for sensitive queries.
Maintain a tested rollback plan.
After an incident
Preserve evidence before remediating.
Identify and close the original point of compromise before rebuilding.
Document findings and update controls to reflect what was learned.
Real-World Incidents, Proofs of Concept, and Research Findings
This section separates confirmed real-world incidents from controlled research, since conflating the two is one of the most common ways this topic gets oversold.
Microsoft's Tay chatbot, March 2016 — documented real-world incident. Tay was designed to learn from conversations with real Twitter users. According to Microsoft's own account, a coordinated group exploited a vulnerability in the bot within the first 24 hours after launch, causing it to post inappropriate content, and Microsoft took it offline shortly afterward. This remains one of the clearest documented cases of a deployed, continuously learning system being manipulated by its own users in production, though it targeted a live conversational feedback loop rather than a formal training pipeline.
Malicious models on the Hugging Face hub, disclosed February 2024 — documented real-world incident. JFrog's security research team found over 100 machine learning models hosted on the platform containing malicious pickle payloads that granted attackers remote shell access when the models were loaded. This is a supply-chain and file-security incident rather than a model-behavior poisoning attack, and it is documented, not hypothetical.
Federated-learning model replacement, Bagdasaryan et al., 2018–2020 — controlled research demonstration. As described earlier, this research showed that a single malicious participant could backdoor a federated model to 100% accuracy on an attacker-chosen task in a single training round, using standard image-classification and word-prediction benchmarks. No claim was made, and none should be inferred, that this specific technique has been observed against a production federated-learning deployment.
Weight poisoning of pretrained NLP models, Kurita, Michel, and Neubig, 2020 — controlled research demonstration. This ACL 2020 paper demonstrated that pretrained weights could carry a backdoor that survives fine-tuning, evaluated on sentiment classification, toxicity detection, and spam detection tasks.
Web-scale dataset poisoning, Carlini and colleagues, 2023 — controlled research demonstration with disclosed practical feasibility. This paper showed that split-view poisoning could have compromised roughly 0.01% of the LAION-400M or COYO-700M datasets for about sixty US dollars at the time of the study, and that frontrunning poisoning could target periodic Wikipedia snapshots. The researchers responsibly disclosed the vulnerability to affected dataset maintainers rather than exploiting it.
Small-sample pretraining backdoors, Anthropic, UK AI Security Institute, and Alan Turing Institute, October 2025 — controlled research demonstration. As discussed above, this study found that 250 malicious documents could backdoor language models between 600 million and 13 billion parameters with a narrow, low-severity trigger behavior. The researchers were explicit that this finding should not be extrapolated to larger frontier models or more harmful behaviors without further study.
PoisonedRAG, Zou, Geng, Wang, and Jia, USENIX Security 2025 — controlled research demonstration. This paper reported a 90% attack success rate injecting as few as five malicious texts per target question into a RAG knowledge base containing millions of documents, and found tested defenses insufficient.
No incident in this list should be read as proof that any specific production model in use today carries a hidden backdoor. The pattern across confirmed real-world cases is different from the pattern in research: confirmed incidents so far have targeted continuously learning feedback loops and supply-chain file security, while the more dramatic training-time and RAG poisoning results remain, as of this writing, demonstrated in controlled research settings.
Limitations, Trade-Offs, and Open Research Questions
Several honest limitations deserve acknowledgment rather than being glossed over.
Detection remains genuinely uncertain for clean-label and semantic-trigger attacks, which are specifically designed to resemble normal data. Adaptive attackers who know which defense a target uses can often adjust their approach to evade it, as Bagdasaryan's own constrain-and-scale technique demonstrated against anomaly detection. Outlier-based defenses trade off against false positives, which can discard legitimate unusual data and degrade a model's coverage of edge cases. Privacy-preserving system designs, including federated learning and differential privacy, inherently limit how closely defenders can inspect the underlying data, which is precisely the tension that makes federated poisoning defenses difficult. Non-independent, non-identically-distributed data across federated participants makes it genuinely hard to distinguish a malicious update from an unusual-but-legitimate one. Model opacity, especially in large neural networks, makes it hard to verify after the fact that no backdoor is present. Continuously learning and agentic AI systems create ongoing feedback loops that can reintroduce poisoning risk even after an initial clean deployment. Supply-chain verification is still maturing as a discipline, and reproducibility across large training runs remains a genuine engineering challenge.
Perhaps the most important limitation to state plainly: there is a real gap between laboratory results, which typically test specific, often narrow behaviors under stated assumptions about attacker access, and confirmed attacks against production systems in the wild. That gap does not mean the research is unimportant; it means secure AI development requires lifecycle-wide controls rather than a single detector or defense, because no current technique claims to close every category of poisoning risk on its own.
FAQ
Are AI model poisoning and data poisoning the same thing?
They overlap heavily but are not identical. Data poisoning specifically manipulates training examples, labels, or feedback data. Model poisoning is the broader category, also covering direct manipulation of model weights, federated-learning updates, or a tampered pretrained checkpoint, without necessarily touching a training dataset at all.
Can ordinary user prompts poison a model?
Not in the technical sense described in this article. A single prompt affects only that one inference-time interaction and does not change the model's underlying weights. Systems that continuously learn from user feedback, however, can be poisoned over time if attackers coordinate to manipulate that feedback signal, as happened with Microsoft's Tay chatbot in 2016.
Does RAG poisoning change the underlying model's weights?
No. RAG poisoning corrupts the external knowledge base or retrieval index a model consults at query time. The base model's parameters are untouched, which is why remediation, such as cleaning or re-indexing the knowledge base, is typically faster than remediating a poisoned model.
What is a model backdoor?
A backdoor is a hidden behavior trained into a model that stays dormant on ordinary inputs and only activates when a specific attacker-chosen trigger appears, such as a phrase, image pattern, or input combination. Because it does not affect normal-case accuracy, standard testing often fails to reveal it.
What is clean-label poisoning?
Clean-label poisoning manipulates only the input features of training examples while keeping their labels technically correct, making the poisoned examples harder to catch through label review than more obvious dirty-label attacks that assign wrong labels outright.
Why is federated learning especially exposed to poisoning?
Federated learning's central privacy benefit, that the aggregator never sees participants' raw data, is also its security weakness: the aggregator can only inspect the model updates participants submit, not the data behind them, which limits how confidently anomalous updates can be distinguished from unusual-but-legitimate ones.
Can a downloaded pretrained model already be poisoned?
Yes. Research on weight poisoning has shown that pretrained weights can carry a backdoor that survives later fine-tuning, and separately, real-world incidents have found malicious code hidden inside model files on public repositories. Both risks call for provenance verification and safe loading practices before a third-party model touches production.
Are larger models more resistant to poisoning?
Not necessarily. A 2025 study found that a fixed, small number of malicious documents, around 250, could backdoor language models regardless of whether they had 600 million or 13 billion parameters, challenging the earlier assumption that bigger models need proportionally more poisoned data to be affected.
How is AI model poisoning detected?
Through layered techniques rather than one tool: data provenance and hash verification, statistical outlier analysis, holdout and trigger-focused testing, checkpoint comparison against known-good baselines, update-norm monitoring in federated systems, and ongoing red-team and monitoring programs. No single method reliably catches every attack type.
Can a poisoned model be repaired?
Sometimes, through techniques such as targeted retraining, weight patching, or rollback to a verified prior version, but repair only works if the original point of compromise, such as a data source, account, or dependency, has also been closed. Retraining on the same compromised pipeline without fixing the root cause tends to reintroduce the same problem.
How common are confirmed AI poisoning attacks in the wild?
Confirmed real-world incidents exist, such as the 2016 Tay chatbot manipulation and the 2024 discovery of malicious models on a public model-hosting platform, but many of the most striking findings in this field, including federated-learning backdoors and small-sample pretraining poisoning, remain demonstrated in controlled research rather than confirmed against production systems. That gap between lab results and field prevalence is itself an open research question.
What is the most important first defensive step?
Establish data and model provenance: know where every training example, dataset, and model checkpoint came from, and be able to verify that with a hash or signature. Nearly every other defense in this article, from outlier detection to incident response, depends on having that baseline in place first.
Key Takeaways
AI model poisoning covers a family of attacks against training data, model parameters, or update pipelines, and is broader than data poisoning alone.
Backdoors are specifically designed to pass normal testing, which is why layered detection matters more than any single check.
Federated learning, LLM pretraining and fine-tuning, RAG knowledge bases, and the AI supply chain each carry distinct poisoning risks that call for different defenses.
A small, fixed number of malicious documents can be enough to backdoor a language model regardless of its overall size, according to 2025 research from Anthropic and its research partners.
RAG poisoning and model poisoning are related but distinct: one corrupts retrieved context, the other corrupts the model's own learned parameters.
Confirmed real-world incidents exist, but many headline findings in this field remain controlled research rather than confirmed field attacks, and the two should not be conflated.
No single safeguard, from robust aggregation to hash verification, fully closes the risk; effective defense requires lifecycle-wide, layered controls.
Incident response after suspected poisoning must close the original point of compromise before rebuilding, or the same attack can recur.
Actionable Next Steps
Inventory every dataset, pretrained model, and third-party dependency currently in production, and record its provenance.
Apply cryptographic hashing or signing to datasets and checkpoints so future tampering can be detected.
Establish an approval workflow for any change to training data, model architecture, or deployment pipeline.
Build or adopt holdout and trigger-focused test sets that specifically probe for anomalous, narrow behaviors.
If using federated learning, evaluate and implement a robust or Byzantine-resilient aggregation method.
If using RAG, schedule a recurring audit of retrieval results for sensitive or high-value queries.
Vet third-party pretrained models before deployment, checking both provenance and file-format safety.
Create a rollback plan and verified known-good baseline for every production model.
Schedule recurring red-team exercises rather than a one-time pre-launch review.
Write an incident response plan specific to suspected poisoning, including who is notified and when legal counsel is engaged.
Glossary
Adversarial machine learning: The study of attacks against machine learning systems and the defenses designed to counter them. See Articsledge's guide to adversarial machine learning.
Availability attack: An attack aimed at broadly degrading a model's overall accuracy or reliability, rather than targeting one specific output.
Backdoor: A hidden behavior trained into a model that stays dormant until a specific attacker-chosen trigger appears.
Checkpoint: A saved snapshot of a model's weights at a given point during or after training.
Clean-label poisoning: A poisoning technique that manipulates only input features while keeping labels technically correct, making it harder to catch through label review.
Data lineage: The traceable history of where a piece of data came from and how it was transformed before reaching a training pipeline.
Data poisoning: Manipulating training, fine-tuning, or feedback data to introduce vulnerabilities, biases, or backdoors. See Articsledge's dedicated guide to AI data poisoning.
Data provenance: Verifiable documentation of a dataset's origin and custody chain.
Dirty-label poisoning: A poisoning technique that deliberately mislabels training examples.
Embedding: A numerical vector representation of text, images, or other data that captures semantic meaning. See Articsledge's guide to word embeddings.
Federated learning: A training approach where many participants jointly build a shared model without exposing their raw local data. See Articsledge's guide to federated learning.
Fine-tuning: Adapting a pretrained model to a specific task using a smaller, targeted dataset. See Articsledge's guide to fine-tuning.
Gradient: A mathematical measure of how much a small change in a model's parameters would change its output error, used to guide training.
Indirect prompt injection: An attack where malicious instructions are embedded in content a model reads, such as a document or web page, rather than typed directly by the user. See Articsledge's guide to prompt injection.
Integrity attack: An attack aimed at making a model produce a specific incorrect output under specific conditions, while otherwise appearing to work normally.
Knowledge-base poisoning: Manipulating the documents or index a retrieval system draws on, without necessarily altering the underlying model. See Articsledge's guide to RAG.
LLM: Large language model, a neural network trained on large volumes of text to generate and understand human language. See Articsledge's guide to large language models.
ML-BOM: A machine learning bill of materials, an inventory of every model, dataset, and dependency used in an AI system, modeled on software bills of materials.
Model poisoning: Manipulating a model's parameters or update process, whether through training data, a malicious federated update, or a tampered checkpoint, so it behaves incorrectly or maliciously.
Model replacement: Substituting an entire malicious model for a legitimate one, often through a compromised distribution or update channel.
Model weights: The numerical parameters a model learns during training that determine its behavior. See Articsledge's guide to model parameters.
Non-IID data: Data that is not independently and identically distributed across participants or sources, common in federated learning and a factor that complicates anomaly detection.
Poison forensics: The process of investigating a suspected poisoning incident to trace it back to its source data, update, or dependency.
Pretraining: The initial, large-scale training phase in which a foundation model learns general patterns from broad data, before any task-specific fine-tuning.
RAG: Retrieval-augmented generation, a technique where a model retrieves external documents at query time to inform its output. See Articsledge's complete guide to RAG.
Robust aggregation: A federated-learning technique designed to reduce the influence of anomalous or malicious participant updates when combining them into a shared model.
Supply-chain attack: A compromise of a shared dependency, such as a dataset, pretrained model, or library, that affects every downstream system relying on it. See Articsledge's guide to AI in the supply chain.
Targeted poisoning: A poisoning attack aimed at manipulating behavior for a narrow, specific set of inputs.
Trigger: The specific condition, such as a phrase, pattern, or input combination, that activates a hidden backdoor behavior.
Untargeted poisoning: A poisoning attack aimed at broadly lowering performance across many inputs, without a specific target output.
Sources & References
Vassilev, A., Oprea, A., Fordyce, A., and Anderson, H. "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations." NIST AI 100-2e2025, National Institute of Standards and Technology, March 24, 2025. csrc.nist.gov/pubs/ai/100/2/e2025/final
National Institute of Standards and Technology. "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile." NIST AI 600-1, July 26, 2024. nist.gov/itl/ai-risk-management-framework
MITRE Corporation. "MITRE ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems." atlas.mitre.org
OWASP GenAI Security Project. "LLM04:2025 Data and Model Poisoning." 2025. genai.owasp.org/llmrisk/llm042025-data-and-model-poisoning
National Security Agency Artificial Intelligence Security Center, Cybersecurity and Infrastructure Security Agency, Federal Bureau of Investigation, and international partners. "AI Data Security: Best Practices for Securing Data Used to Train and Operate AI Systems." May 22, 2025. cisa.gov/news-events/alerts/2025/05/22/new-best-practices-guide-securing-ai-data-released
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., and Shmatikov, V. "How To Backdoor Federated Learning." Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020. arxiv.org/abs/1807.00459
Kurita, K., Michel, P., and Neubig, G. "Weight Poisoning Attacks on Pretrained Models." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 2793 to 2806. aclanthology.org/2020.acl-main.249
Anthropic, UK AI Security Institute, and Alan Turing Institute. "A Small Number of Samples Can Poison LLMs of Any Size." Anthropic Research, October 9, 2025. anthropic.com/research/small-samples-poison
Hubinger, E., et al. "Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training." Anthropic, 2024. anthropic.com/research/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training
Zou, W., Geng, R., Wang, B., and Jia, J. "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models." Proceedings of the 34th USENIX Security Symposium, 2025, pp. 3827 to 3844. usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramer, F. "Poisoning Web-Scale Training Datasets Is Practical." 2023. arxiv.org/abs/2302.10149
Cohen, D. "Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor." JFrog Security Research, February 2024, as reported in The Hacker News, March 4, 2024. thehackernews.com/2024/03/over-100-malicious-aiml-models-found-on.html
Lee, P. "Learning From Tay's Introduction." The Official Microsoft Blog, March 25, 2016. blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction