What is Adversarial machine learning?
- 3 days ago
- 30 min read
A self-driving car reads a stop sign as a speed limit sign. A spam filter waves through a phishing email disguised with a few invisible characters. A chatbot connected to company files quietly leaks a customer list because a poisoned document told it to. These are not ordinary bugs. They are adversarial machine learning: the deliberate manipulation of AI systems, and the defenses against it.
TL;DR
Adversarial machine learning (AML) is the study of attacks that intentionally fool, poison, or extract information from machine learning and generative AI systems, plus the defenses against them.
NIST's 2025 taxonomy (NIST AI 100-2e2025) splits AML into predictive-AI risks (evasion, poisoning, privacy attacks) and generative-AI risks (prompt injection, RAG poisoning, agent misuse).
Attacks range from tiny, invisible changes to an image to poisoned training data, stolen models, and malicious instructions hidden inside documents an AI reads.
No defense makes a model completely secure. Adversarial training, input filtering, and monitoring reduce risk; none eliminate it.
Generative AI and AI agents extend the field beyond pixels and classifiers into prompts, retrieved documents, and tool calls.
A practical AML program combines threat modeling, secure MLOps, layered defenses, and honest, adaptive evaluation, not a single tool.
What is Adversarial machine learning?
Adversarial machine learning is the branch of AI security that studies how attackers deliberately manipulate machine learning systems, through crafted inputs, poisoned data, or malicious instructions, to cause misclassification, leak private information, or produce unsafe outputs, and it studies the defenses that reduce those risks across a model's full lifecycle.
Table of contents
What Is Adversarial Machine Learning?
Adversarial machine learning is the study of how someone can deliberately manipulate a machine learning system, and how defenders can reduce that risk. The word "adversarial" points to intent. A person chooses an input, a piece of training data, or a message specifically because it will make the model fail in a useful way for the attacker.
The U.S. National Institute of Standards and Technology (NIST) defines and organizes this field in its report Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2e2025), published March 24, 2025. NIST splits AI systems into two broad classes: predictive AI, which classifies, scores, or forecasts, and generative AI, which produces text, images, or other content (NIST, 2025).
It helps to separate deliberate adversarial manipulation from four things it is not:
Ordinary model errors: a model can misclassify an image or hallucinate a fact without anyone attacking it. That is a performance problem, not an adversarial one.
Model drift: real-world data changes over time (sometimes called concept drift or data drift), and accuracy can fall without any attacker involved.
Random noise: sensor noise, compression artifacts, or blur can confuse a model, but nobody chose that noise to cause a specific failure.
Naturally hard examples: some inputs are genuinely ambiguous, even to humans. A blurry photo of a dog that looks like a wolf is difficult, not adversarial.
The distinguishing feature of an adversarial example or attack is intent and optimization: an attacker searches for the input, data point, or instruction that most efficiently breaks the system's integrity, confidentiality, or availability.
Why Adversarial Machine Learning Matters
Machine learning now sits inside decisions that affect money, safety, and privacy. When those systems can be manipulated, the consequences extend beyond a wrong prediction.
Financial risk: fraud-detection and credit-scoring models can be probed and evaded, letting fraudulent transactions or applications slip through undetected.
Safety: computer-vision systems in autonomous vehicles and medical imaging can be fooled by crafted stickers, patches, or pixel changes, as documented in NIST AI 100-2e2025's discussion of real-world failures including vehicles swerving after sign misclassification.
Privacy: membership inference and model inversion attacks can expose whether a specific person's data was used to train a model, or reconstruct sensitive attributes from it.
Content integrity: malware-detection and content-moderation classifiers can be evaded by attackers who slightly alter malicious files or text.
Trust and governance: an organisation that ships an AI feature without threat modeling it risks reputational damage, regulatory scrutiny, and loss of user trust when a manipulation succeeds publicly.
Generative AI adds a new layer of exposure. A customer-facing chatbot or an internal assistant that reads emails, documents, or web pages can be manipulated through the content it reads, not just through direct user input. This is why frameworks like MITRE ATLAS and the OWASP Top 10 for LLM Applications now sit alongside NIST's taxonomy as reference points for security teams.
A Brief History of Adversarial Machine Learning
Adversarial machine learning did not begin with deep learning. Security researchers were already studying how spam filters and intrusion-detection systems could be gamed in the 2000s, well before today's neural networks existed.
2004–2010: early work on adversarial classification and spam-filter evasion established that any learned decision boundary can, in principle, be probed and manipulated by a motivated attacker.
2013–2014: Szegedy et al.'s paper "Intriguing Properties of Neural Networks" (ICLR, 2014) showed that state-of-the-art image classifiers, including networks trained on ImageNet, could be fooled by small, often imperceptible pixel changes. This introduced the term "adversarial example" to the deep-learning community.
2015: Goodfellow, Shlens, and Szegedy published "Explaining and Harnessing Adversarial Examples" (ICLR, 2015), introducing the Fast Gradient Sign Method (FGSM) and arguing that the linear behaviour of high-dimensional models, not just their nonlinearity, explains their vulnerability.
2016: Microsoft's Tay chatbot was manipulated by coordinated Twitter users into producing offensive content within 24 hours of launch, a widely cited real-world case of data poisoning through live user interaction, documented in MITRE ATLAS's case study library.
2017–2018: Madry et al.'s "Towards Deep Learning Models Resistant to Adversarial Attacks" (2018) formalized projected gradient descent (PGD) as a strong first-order attack and adversarial training as a defense. Carlini and Wagner published optimization-based attacks (2017) that became a standard benchmark for evaluating defenses.
2018–2020: researchers demonstrated physical-world adversarial examples, including stickers on road surfaces and modified road signs that fooled traffic-sign classifiers and, in controlled tests, driver-assistance systems.
2019: NIST published its first taxonomy draft efforts culminating in the original AI 100-2e2023 report, standardising terminology across the field.
2021: MITRE launched ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems), an ATT&CK-style knowledge base of real-world and demonstrated attacker techniques against AI systems.
2023–2025: the rapid adoption of large language models expanded the field into prompt injection, jailbreaking, retrieval-augmented generation (RAG) poisoning, and AI-agent manipulation. NIST's March 2025 update (AI 100-2e2025) and MITRE ATLAS's 2025–2026 releases both added dedicated coverage for these generative-AI and agentic risks.
This history matters because it shows that adversarial machine learning is not a passing side effect of deep learning. It is a persistent property of any system that learns from data and makes decisions that someone else has an incentive to influence.
How an Adversarial Attack Works
Most adversarial attacks share the same basic shape. An attacker picks a goal, works within some set of allowed actions, and uses whatever knowledge and access they have to search for the input, data point, or message that best achieves that goal.
Objective: what the attacker wants. Common objectives are causing a wrong prediction (evasion), corrupting a model during training (poisoning), stealing a model's behaviour (extraction), or exposing training data (privacy attack).
Allowed actions: what the attacker can actually change. This might be a handful of pixels, a few words in a document, or a small number of poisoned training rows.
Constraints: limits that keep the attack realistic, such as staying visually unnoticeable, staying within a query budget, or staying physically producible (like a printable sticker).
Knowledge: how much the attacker knows about the model, covered in detail in the threat-model section below.
Success criteria: how the attacker (or a defender testing the system) measures whether the attack worked.
A harmless way to picture this: imagine a simple model that approves or denies a loan application based on a handful of numeric features. An attacker who knows which features matter most could nudge a borderline application's numbers just enough to flip the decision, without changing the applicant's real financial position. The attacker is not guessing randomly. They are searching, often using the model's own feedback (approved or denied) to guide each nudge. This search-and-optimize pattern, not any specific formula, is the core idea behind most adversarial techniques, from simple probing to gradient-based image attacks.
Understanding the Threat Model
A defense is only meaningful relative to a clearly stated threat model. Calling a model "robust" without saying against whom is not a meaningful claim.
Attacker Knowledge: White-Box, Grey-Box, and Black-Box
White-box: the attacker has full access to the model's architecture, parameters, and gradients. This is the strongest assumption and is used to test worst-case robustness.
Grey-box: the attacker has partial knowledge, such as the model family or training data distribution, but not exact parameters.
Black-box: the attacker can only query the model and observe outputs (labels or scores). Black-box attacks are still effective, often by training a substitute model and exploiting transferability, or by using many queries to estimate gradients.
Attacker Goals: Targeted vs. Untargeted, and the CIA Triad
An untargeted attack just wants a wrong answer. A targeted attack wants a specific wrong answer, such as making a stop sign read as a speed-limit sign. Attacker goals also map onto classic security properties:
Integrity: causing incorrect outputs (most evasion and poisoning attacks).
Availability: degrading the system so it cannot serve legitimate users, for example through resource-exhaustion queries.
Confidentiality and privacy: extracting the model itself or information about its training data.
Training-Time vs. Inference-Time, and Digital vs. Physical
Training-time attacks (poisoning, backdoors) corrupt the model before or during learning. Inference-time attacks (evasion) target an already-deployed model. Separately, digital attacks modify a file directly, while physical attacks must survive real-world conditions such as lighting, camera angle, and printing, which is a much harder constraint for an attacker to satisfy reliably.
Query budgets and rate limits also matter. A black-box attacker who needs a million queries to succeed is a very different risk than one who needs ten. This is why query-budget assumptions belong in every threat model and every defense evaluation.
The Machine-Learning Lifecycle Attack Surface
Adversarial risk is not limited to the moment a model makes a prediction. Every stage of the machine learning lifecycle introduces its own way for a security failure to enter.
Lifecycle Stage | Typical Adversarial Risk |
|---|---|
Data collection | Poisoned or scraped data from untrusted sources; web-scraped training corpora with planted content |
Data labelling | Malicious or careless labellers introduce mislabeled examples (dirty-label poisoning) |
Data storage | Unauthorized access lets an insider or attacker tamper with stored datasets |
Feature engineering | Feature pipelines can be manipulated to hide malicious signals from a model |
Training | Poisoning, backdoor triggers, and hyperparameter or optimizer manipulation |
Third-party models and dependencies | Supply-chain risk from pretrained models, plugins, or open-source packages |
Model evaluation | Weak or non-adaptive evaluation creates false confidence in robustness |
Deployment | Misconfigured APIs expose confidence scores that assist black-box attacks |
APIs and user interfaces | Direct evasion inputs and, for generative systems, prompt injection |
Monitoring | Missing drift or abuse detection lets manipulation continue unnoticed |
Model updates | Continuous learning pipelines can be poisoned through live user feedback, as in the Tay incident |
Retirement | Decommissioned models or leftover endpoints left reachable can still be probed or extracted |
Treating adversarial risk as only an input-filtering problem misses most of this surface; a strong programme reviews every lifecycle stage.
Major Types of Adversarial Machine-Learning Attacks
NIST AI 100-2e2025 and MITRE ATLAS both organize predictive-AI attacks into a handful of recurring categories. Each is explained below with what it is, when it occurs, the attacker's likely goal, a safe illustrative example, its potential impact, and the main defensive considerations.
Evasion and Adversarial-Example Attacks
Evasion happens at inference time, after training is complete. The attacker crafts an input designed to be misclassified while looking normal to a human or an automated pre-filter. The goal is usually to slip past a filter, such as spam, malware, or fraud detection. A safe example: a spam email with a few extra spaces or homoglyph characters that a text classifier has not learned to ignore. Impact ranges from nuisance to serious safety and financial harm. Defensively, adversarial training, input preprocessing, and ensemble detection all help, but none guarantee success against an adaptive attacker.
Data Poisoning
Poisoning happens at training time. The attacker inserts or alters training examples so the model learns a distorted pattern. The goal can be indiscriminate (lower overall accuracy) or targeted (make one specific input class behave wrongly). A safe example: submitting many slightly-mislabelled reviews to a sentiment classifier's public training feed so it slowly misjudges a specific product category, related to the data quality problems every ML pipeline must guard against. Impact includes silently degraded accuracy and hard-to-detect targeted failures. Defensively, data provenance checks, outlier detection, and holding back a trusted validation set help limit exposure.
Model Poisoning
Model poisoning modifies the model artifact or its training process directly, rather than only the data, for example through a compromised training pipeline, a malicious contributor in federated learning, or tampered checkpoints. The goal is usually to implant a persistent, hard-to-detect flaw. Impact can be severe because the corruption lives inside the model weights themselves. Defensively, signed model artifacts, reproducible training, and independent re-training checks are the main mitigations.
Backdoor and Trojan Attacks
A backdoor is a hidden trigger, such as a small visual pattern or a specific phrase, planted during training so the model behaves normally until that trigger appears, then produces an attacker-chosen output. The goal is stealth combined with reliable, on-demand control. A safe illustrative example: a poisoned training set teaches an image classifier that any photo containing a small yellow sticker should be labelled "authorized," regardless of the rest of the image. Impact is high because clean-looking test performance hides the flaw. Defensively, trigger-detection tools, activation-clustering analysis, and careful vetting of third-party or pretrained models are the standard responses.
Model Extraction or Model Stealing
Extraction attacks query a model repeatedly and use the input-output pairs to train a substitute model that mimics the original, without ever seeing its parameters. The goal is usually to steal intellectual property or to build a local copy for offline attack development. Impact includes lost competitive advantage and easier follow-on attacks, since the stolen substitute can be probed offline without rate limits. Defensively, query rate limiting, output rounding, and watermarking model outputs all raise the cost of extraction.
Model Inversion
Model inversion attempts to reconstruct representative training inputs, such as an approximate face image, from a model's outputs or gradients. The goal is to recover sensitive information the model was never meant to reveal directly. Impact is a privacy breach, especially for models trained on biometric, medical, or personal data. Defensively, differential privacy during training and limiting the granularity of returned confidence scores both reduce (but do not eliminate) this risk.
Membership Inference
Membership inference asks a narrower question: was a specific record part of the training data at all? An attacker compares how confidently a model responds to a known record versus unseen data. The goal is often to confirm someone's sensitive data (for example, that they were a patient in a specific study) was used to train a model. Impact is a privacy and, in regulated sectors, a compliance concern. Defensively, differential privacy, regularization, and limiting overfitting all reduce the confidence gap that these attacks exploit.
Privacy Leakage
Beyond inversion and membership inference, models (especially generative ones) can leak fragments of their training data directly in outputs, such as verbatim text a language model saw many times during training. The goal may not even require a targeted attack; leakage can surface from ordinary use. Impact includes exposure of confidential documents or personal data embedded in training corpora. Defensively, training-data de-duplication, output filtering, and differential privacy training are the main levers.
Availability and Resource-Exhaustion Attacks
These attacks aim to make a system unusable rather than wrong, for example by sending inputs specifically crafted to maximize a model's computation time, or by flooding an API with expensive queries. The goal is denial of service or inflated operating costs, sometimes called "denial of wallet" for pay-per-use AI services. Impact includes downtime and unexpected cloud bills. Defensively, rate limiting, timeouts, and input-complexity caps are standard controls.
AI Supply-Chain Attacks
Modern ML systems depend on pretrained models, open-source libraries, datasets, and plugins from outside the organisation. A supply-chain attack compromises one of these upstream components so the compromise spreads to every downstream user. The goal is broad, often opportunistic compromise. Impact can be very large because a single tampered package or model checkpoint can affect thousands of downstream deployments. Defensively, verifying model and package provenance, checksums, and maintaining a software/AI bill of materials all help contain this risk.
Adversarial Examples in More Detail
Adversarial examples are the most studied and most misunderstood corner of the field. They deserve a closer look, with an honest note that pixel-level image attacks are only one part of a much bigger picture.
A perturbation is the small change added to an input. Researchers describe how "small" a perturbation is using distance metrics such as the L2 (Euclidean) or L-infinity (maximum single-pixel change) norm. A perturbation that is small under one of these norms is not automatically unnoticeable to a human, and it is not automatically realistic in the physical world; this is the difference between mathematical imperceptibility and operational realism that the field has learned to treat separately.
FGSM (Fast Gradient Sign Method): a single-step attack introduced by Goodfellow, Shlens, and Szegedy (2015) that nudges every input value slightly in the direction that most increases the model's error, based on the sign of the loss gradient.
PGD (Projected Gradient Descent): an iterative refinement of FGSM, formalized by Madry et al. (2018), that takes several small steps and "projects" the result back within an allowed perturbation budget after each step, producing a substantially stronger attack.
Carlini-Wagner (C&W) attacks: optimization-based attacks (Carlini and Wagner, 2017) that directly minimize the size of the perturbation needed to cause misclassification, and remain one of the standard benchmarks for evaluating whether a proposed defense actually works.
Transferability: an adversarial example crafted against one model often fools a different model trained on similar data, even without access to that second model's parameters. This is what makes black-box attacks practical.
Universal perturbations: a single, fixed perturbation pattern that causes misclassification across many different input images, rather than being tailored to just one.
Physical adversarial examples: perturbations realized in the physical world rather than only in a digital file, such as printed patches or road stickers. Researchers from Keen Security Lab and McAfee separately demonstrated that stickers or tape on road markings and speed-limit signs could cause commercial driver-assistance systems to misread lane position or speed limits in controlled tests.
None of this means pixel attacks are the whole story. The same underlying idea, searching for a small, targeted change that flips a model's decision, reappears in text (changing a few words to evade a spam filter), audio (adding inaudible noise to fool speech recognition), and now prompts (changing a few words to bypass a language model's safety training). Treating "adversarial examples" as only an image-classifier problem badly understates the field.
Poisoning, Backdoors and Training-Time Manipulation
Poisoning and backdoor attacks deserve their own closer look because they behave very differently from evasion: the damage is done once, during training, and can persist silently for the model's entire operational life.
Clean-label poisoning: poisoned examples keep their correct-looking label, making the tampering far harder to spot through manual review than examples with obviously wrong labels.
Dirty-label poisoning: poisoned examples carry an incorrect label outright, which is easier to catch with basic label-consistency checks, but still effective if review is weak or automated.
Targeted vs. indiscriminate poisoning: targeted poisoning aims to corrupt behaviour for one specific input or class; indiscriminate poisoning aims to degrade overall accuracy.
Backdoor triggers: a specific pattern (a pixel patch, a rare word, a particular sensor reading) that the model has been trained to associate with an attacker-chosen output, while behaving normally otherwise.
Outsourced-training risk: training a model on rented infrastructure or through a third-party service introduces a party who could tamper with the process, not just the data.
Public-data risk: scraping training data from the open web means an attacker can seed the web with content specifically designed to be scraped and to corrupt future models, a documented concern for large web-trained corpora.
Model-update risk: systems that continue learning from live user feedback after deployment, as Microsoft's Tay chatbot did, can be poisoned incrementally by user interactions rather than by a one-time training-data compromise.
It is worth being precise: poisoning corrupts the data a model learns from, while directly modifying a saved model's weights or code corrupts the artifact itself. Both can produce hidden, triggerable misbehaviour, but they need different defenses, data provenance for the former, artifact integrity checks for the latter.
Adversarial Machine Learning in Generative AI
Large language models and multimodal generative systems inherit the predictive-AI risks above and add new ones. The 2025 OWASP Top 10 for LLM Applications and NIST AI 100-2e2025 both treat this as an extension of adversarial machine learning, not a separate field, though the two overlap only partly.
Direct prompt injection: a user writes instructions specifically designed to override a system's intended behaviour, such as asking it to ignore its safety instructions.
Indirect prompt injection: malicious instructions are hidden inside content the model reads, such as a web page, PDF, or email, rather than typed directly by the user. This is the vector behind documented incidents, described below, where a single crafted document misled production assistants.
Jailbreaking (conceptual level): techniques that attempt to get a model to bypass its safety training, often through role-play framing or incremental escalation. This guide explains the concept for defensive awareness without providing working jailbreak text.
System-prompt extraction: attempts to make a model reveal its own confidential system instructions.
Training-data and fine-tuning poisoning: the same poisoning risks described earlier apply to the large web-scraped corpora and fine-tuning datasets used for language models.
RAG or index poisoning: planting misleading or malicious documents inside a knowledge base that a retrieval-augmented generation system will later fetch and treat as trustworthy context.
Data leakage: a model reproducing sensitive text it saw during training or retrieval, whether through direct memorization or through careless context handling.
Tool and plugin manipulation: tricking a model into misusing the external tools or APIs it has been given access to, such as sending data to an unintended recipient.
Malicious retrieved content: documents or web pages that are not just misleading but contain embedded instructions meant to hijack the model's next action.
Excessive agency: giving a model or agent more autonomous permission (to send emails, execute code, move money) than its trust level and oversight actually justify.
Model extraction concerns: the same stealing risk described for predictive models applies to proprietary language models exposed through an API.
Multimodal adversarial inputs: adversarial content hidden in images, audio, or video that a multimodal model processes alongside text.
Generative-AI security genuinely extends beyond classic AML: classic systems assumed a fixed input-output classifier, while generative systems have open-ended outputs, call external tools, and retrieve untrusted content live, which is why OWASP treats excessive agency and supply-chain risk as first-class concerns. See also this site's guide to AI jailbreaking for a closer look at that specific technique.
Adversarial Risks in RAG and AI Agents
Retrieval-augmented generation, explained in more depth in this site's guide to naive RAG, lets a language model pull in outside documents at answer time instead of relying only on what it learned during training. AI agents go further, letting a model take actions such as browsing, emailing, or running code. Both extend the model's attack surface to include everything it reads and every tool it can call.
Poisoned knowledge sources: an attacker plants misleading or malicious content in a document repository, wiki, or shared drive the system will later retrieve. Security researchers at Zenity Labs demonstrated in 2025 that a single crafted document could reliably mislead a production Microsoft 365 Copilot deployment into giving false answers.
Malicious instructions embedded in retrieved documents: text formatted to look like part of the document's content but written to be read as an instruction by the model.
Retrieval manipulation: gaming the retrieval system itself, for example by stuffing a document with terms likely to make it rank highly for many unrelated queries, so its (poisoned) content keeps surfacing.
Trust-boundary failures: treating retrieved or tool-returned content with the same trust as a direct user instruction, instead of treating it as untrusted data.
Unsafe tool calls: an agent invoking a tool (sending an email, deleting a file, making a purchase) based on manipulated instructions it picked up from content it processed.
Memory poisoning: corrupting an agent's persistent memory or notes so future sessions inherit a manipulated state.
Permission design, least privilege, and human approval: giving an agent only the access it needs for its task, and requiring a human to approve high-impact actions, are the most effective structural defenses.
Sandboxing: running agent actions in an isolated environment so a manipulated tool call cannot reach production systems directly.
Traceability and output validation: logging what an agent retrieved and why it acted, and validating outputs before they trigger real-world effects.
This guide will not provide working attack payloads. Understanding these techniques means designing systems where one poisoned document or manipulated tool call cannot cause outsized harm.
Real-World Examples and Research Demonstrations
Precision matters here. Each example below is labelled by what kind of evidence it actually represents, since conflating a lab demonstration with a widespread production incident is one of the most common errors in AML reporting.
Microsoft's Tay chatbot (2016) — documented real-world incident: coordinated Twitter users fed Tay abusive content, and its live-learning design incorporated that content into its responses within about 16 hours, forcing Microsoft to shut it down. MITRE ATLAS catalogs this as case study AML.CS0009. It teaches that any system that learns from live, unfiltered user interaction needs strong input moderation before launch, not after.
Physical adversarial patches on road signs — controlled research demonstrations: researchers, including teams from Keen Security Lab and McAfee Advanced Threat Research, showed in controlled tests that stickers or tape on road signs could cause specific driver-assistance systems to misread lane markings or speed limits. These are lab and closed-course demonstrations, not documented mass real-world attacks on public roads, and they show that physical-world robustness is a genuinely different, harder problem than digital robustness.
RAG and Copilot-style indirect prompt injection — documented real-world and vendor-disclosed demonstrations: security researchers (including Zenity Labs and Noma Security) have separately shown that a single crafted document or shared file could mislead production retrieval-augmented systems into false answers or unintended data exposure, illustrating that indirect prompt injection is a present, not hypothetical, risk for RAG-connected assistants.
Academic model-extraction demonstrations — controlled research: researchers have repeatedly shown, in lab settings, that querying a hosted model enough times lets an attacker train a substitute model with similar behaviour, informing why rate limiting and query monitoring are now standard advice rather than optional hardening.
These examples do not prove every deployed model is easily attacked in production. A lab demonstration does not automatically transfer to every real deployment with its own defenses and constraints.
How Organisations Defend Machine-Learning Systems
No single control secures a machine learning system. Effective defense works in layers, so that if one control fails, another still limits the damage.
Governance and threat modeling: identify what could go wrong for this specific system, before choosing controls, rather than applying a generic checklist.
Data provenance and validation: track where training data came from and screen it for anomalies before it reaches a training run.
Access control: restrict who and what can modify training data, model artifacts, and deployment configuration.
Secure development: apply standard secure software practices (code review, dependency scanning) to the ML codebase itself, not just the model.
Supply-chain security: verify the provenance of pretrained models, datasets, and packages before use.
Model-level defenses: adversarial training and certified-robustness techniques, covered in the next section.
Input and output controls: validate inputs for anomalies and filter or review outputs before they reach users or downstream systems.
API security: rate-limit queries, avoid exposing raw confidence scores, and authenticate callers.
Runtime isolation: sandbox model execution and any tools an AI agent can call, so a manipulated action cannot directly reach production systems.
Monitoring and logging: track input distributions, output patterns, and query behaviour over time.
Human oversight: require human review for high-impact or ambiguous decisions.
Incident response: have a plan for containing and rolling back a compromised model before an attack is confirmed, not after.
Recovery and model rollback: keep a known-good previous model version ready to restore quickly.
Adversarial Training and Robust Optimisation
Adversarial training is the most widely used model-level defense against evasion. During training, the model is repeatedly shown adversarial examples, generated using an attack such as PGD, alongside clean data, and penalized for misclassifying them. Over many rounds, this pushes the model toward decision boundaries that are harder to exploit with that class of attack.
Strengths: adversarial training reliably improves robustness against the specific attack (and perturbation budget) it was trained on, and remains one of the few defenses that has consistently survived independent adaptive re-evaluation.
Computational cost: generating adversarial examples during every training step is expensive, often several times the cost of standard training, which limits its use at very large scale.
Dependence on the training threat model: a model adversarially trained against one attack type or perturbation size is not automatically robust against a different attack or a larger perturbation budget.
Not a universal solution: adversarial training typically trades some clean-data accuracy for robustness, and it has not been shown to generalize perfectly to attacks the defender did not anticipate.
It helps to separate two related but different goals.
Empirical robustness: measured by testing against known attacks; a model can look robust simply because it was tested against weak or non-adaptive attacks (see the evaluation section below).
Certified robustness: mathematically guarantees that no perturbation within a defined budget can change the model's output, using techniques such as randomized smoothing or convex relaxation. Certified guarantees are stronger but usually apply to smaller perturbation budgets and can be more computationally demanding than empirical adversarial training.
Detection, Monitoring and Incident Response
Because no defense is complete, detecting an attack in progress, or shortly after, matters as much as preventing one.
Input-distribution monitoring: flag inputs that look statistically different from what the model normally sees.
Behavioural anomalies: watch for unusual patterns in how a system is used, not just individual inputs.
Query patterns: a burst of similar, systematically varied queries can indicate model-extraction or black-box attack probing.
Model-output changes: sudden shifts in output distribution can indicate poisoning, drift, or an active attack.
Drift detection: separate ,[object Object], and ,[object Object], monitoring from adversarial-attack detection; both can look similar but need different responses.
Canary tests: run known test cases regularly to catch unexpected behaviour changes quickly.
Abuse detection and rate limits: throttle and flag accounts or IPs generating unusual volumes of queries.
Logging with privacy safeguards: log enough to investigate incidents without over-retaining sensitive user data.
Alert triage, reproduction, and containment: confirm a suspected attack is real before acting, then isolate the affected component.
Model rollback: restore a known-good model version while investigating a suspected poisoning or compromise.
Post-incident learning: feed findings back into threat models and monitoring rules.
Reliably detecting a genuinely novel adversarial input remains hard. Monitoring reduces risk; it does not guarantee detection.
Evaluating Adversarial Robustness Correctly
A surprising number of published defenses have later been shown to fail against a slightly stronger or better-informed attacker. Evaluating robustness correctly is as important as building the defense itself.
Clean accuracy vs. robust accuracy: report both; a defense that only reports clean accuracy is not reporting robustness at all.
Clearly defined threat models: state the exact attack, perturbation budget, and attacker knowledge assumed.
Strong baselines: compare against well-known, strong attacks (PGD, C&W), not only weak or outdated ones.
Adaptive attackers: test against an attacker who knows the defense exists and is specifically designed to defeat it, not just a generic attack.
White-box evaluation: test under the strongest reasonable assumption, since weaker assumptions can hide real weaknesses.
Attack-parameter reporting: publish exact settings used, so results can be reproduced and checked.
Avoiding gradient masking: a well-known failure mode (documented by Athalye, Carlini, and Wagner, 2018, in "Obfuscated Gradients Give a False Sense of Security") where a defense appears robust only because it makes gradients hard to compute, not because the underlying vulnerability is fixed.
Independent red teaming: have a separate team, without a stake in the defense's success, attempt to break it.
Reproducibility: share code and exact settings so others can verify claimed results.
Testing realistic end-to-end systems: evaluate the full application, not just the isolated model, since surrounding code often introduces its own weaknesses.
Evaluating both model and surrounding application: a hardened model behind a leaky API is still an insecure system.
Weak attacks create a false sense of security: resisting basic FGSM but never PGD or an adaptive attacker proves little.
Secure MLOps Checklist
Before Training
Document data sources and verify provenance for every training dataset.
Screen data for obvious label inconsistencies and statistical outliers.
Define the threat model for this specific system before choosing controls.
During Training
Hold out a trusted, independently sourced validation set that was never exposed to public contribution.
Log training data lineage and hyperparameters for reproducibility.
Where feasible, apply adversarial training or certified-robustness techniques for the intended threat model.
Before Deployment
Run adaptive, independent red-team evaluation before launch, not only automated benchmark scores.
Rate-limit and authenticate API access; avoid exposing raw confidence scores unnecessarily.
Verify the provenance of any third-party or pretrained model components.
During Operation
Monitor input distributions, query patterns, and output behaviour continuously.
Keep a known-good model version ready for fast rollback.
Review and limit the permissions of any connected tools or agents on a regular schedule.
After an Incident
Contain first: isolate or roll back the affected component before full root-cause analysis.
Reproduce the issue in a controlled environment to confirm the attack vector.
Update threat models, monitoring rules, and training data screening based on what was learned.
Common Misconceptions
"Adversarial machine learning only affects image classifiers." Poisoning, extraction, membership inference, and prompt injection all apply well beyond computer vision, including to text, audio, tabular data, and generative systems.
"Every model mistake is an adversarial attack." Ordinary errors, drift, and naturally hard examples are common and are not evidence of an attack unless intent and optimization can be shown.
"Small perturbations are always invisible." A perturbation can be mathematically small under a chosen distance metric while still being visible, or conversely, be visible-looking but still count as "small" by that metric. Imperceptibility and metric size are related but not identical.
"Adversarial training makes a model secure." It measurably improves robustness against the attacks it was trained on; it does not make a model universally secure against all attacks, forever.
"A black-box attacker cannot attack a model." Black-box attacks are well documented, using query-based estimation or transferability from a substitute model.
"Prompt injection and traditional adversarial examples are identical." They share the underlying idea of a crafted input causing unintended behaviour, but prompt injection exploits instruction-following in language models and often arrives through content the system reads, not through gradient-based pixel search.
"High test accuracy means the model is secure." Standard test accuracy measures performance on typical data, not resistance to a motivated adversary; the two are evaluated differently, as covered above.
"One security tool can solve the entire problem." Defense in depth, across the full lifecycle, is the consistent recommendation from NIST, MITRE ATLAS, and OWASP alike; no single control is sufficient.
Open Problems and Future Directions
Several important questions in adversarial machine learning remain genuinely unresolved, and the field is honest about that.
Robustness–accuracy–cost trade-offs: gaining strong robustness without raising cost or reducing clean accuracy remains unsolved in most settings.
Evaluation standards: the field still lacks one universally agreed benchmark for measuring robustness fairly.
Adaptive attacks: new defenses keep getting broken by adaptive attackers, showing static evaluation alone is never enough.
Semantic and physical realism: connecting perturbation budgets to what is achievable in the physical world is still active research.
Foundation-model scale: adversarial training that works on smaller models does not always scale cleanly to today's largest models.
Multimodal and RAG/agent systems: these newer attack surfaces have less mature defenses than classic evasion and poisoning.
Continual learning and privacy: systems that keep learning after deployment face ongoing poisoning risk, and differential-privacy guarantees still trade off against model utility.
Benchmarks and terminology: many published benchmarks do not reflect real deployment conditions, and NIST, MITRE, and OWASP terminology is not yet fully unified.
FAQ
What is adversarial machine learning in simple terms?
It is the study of how people can deliberately trick, corrupt, or steal information from AI systems, and how to defend against that. It covers everything from small changes to an image that fool a classifier to poisoned training data and malicious instructions hidden in documents an AI reads.
What is an adversarial attack in machine learning?
An adversarial attack is any deliberate action, crafting an input, poisoning training data, or manipulating a prompt, designed to make a machine learning system misbehave in a way that benefits the attacker, whether that means a wrong prediction, a data leak, or an unsafe output.
What is an adversarial example?
An adversarial example is an input, often only slightly different from a normal one, that is specifically crafted to cause a model to make an error. In image classification, this can be a small pixel change invisible to most people; in text or prompts, it can be a few carefully chosen words.
How do adversarial attacks work?
Most attacks follow the same pattern: the attacker picks a goal (like causing a misclassification), works within some limited set of allowed changes, and searches for the specific input or data change that best achieves that goal, often using the model's own feedback to guide the search.
What is the difference between evasion and poisoning?
Evasion happens at inference time, after training, and targets an already-deployed model with a crafted input. Poisoning happens at training time, corrupting the data or process the model learns from, so the damage is baked into the model before it is ever deployed.
Is prompt injection a form of adversarial machine learning?
Yes, most security frameworks, including NIST's 2025 taxonomy and OWASP's LLM Top 10, treat prompt injection as part of adversarial machine learning applied to generative AI, though it works differently from classic pixel-based adversarial examples since it exploits instruction-following behaviour rather than gradient-based optimization.
What are white-box, grey-box, and black-box attacks?
These describe how much the attacker knows. White-box attackers have full access to model parameters and gradients. Grey-box attackers have partial knowledge, such as the model family. Black-box attackers can only send queries and observe outputs, yet can still succeed using techniques like transferability.
How can organisations defend machine-learning systems?
Through layered defenses across the entire ML lifecycle: strong data provenance, adversarial training where feasible, input and output validation, API rate limiting, continuous monitoring, human oversight for high-impact decisions, and a tested incident-response and rollback plan.
Does adversarial training solve the problem?
No. Adversarial training measurably improves robustness against the specific attacks it was trained against, but it comes with real computational cost, often reduces clean-data accuracy somewhat, and does not guarantee robustness against attacks the defender did not anticipate.
How should adversarial robustness be evaluated?
By reporting both clean and robust accuracy under a clearly defined threat model, testing against strong and adaptive attacks (not just weak baseline attacks), checking for gradient masking, and ideally having an independent red team try to break the defense before trusting the results.
What is the difference between AI security and adversarial machine learning?
AI security is the broader discipline covering all risks to AI systems, including conventional cybersecurity issues like access control and infrastructure security. Adversarial machine learning is the more specific field focused on attacks that exploit how a model learns and makes decisions.
Can a black-box attacker really attack a model they cannot see inside?
Yes. Black-box attackers commonly train a substitute model on observed input-output pairs and exploit transferability, or use repeated queries to estimate how the target model responds to different inputs, without ever seeing its internal parameters.
What is data poisoning and how is it different from a normal data-quality problem?
Data poisoning is a deliberate insertion of misleading or malicious training examples designed to corrupt a model's behaviour in a specific way. An ordinary data-quality problem, like accidental mislabeling, is unintentional and usually spread randomly rather than targeted at a specific outcome.
Is every unexpected AI output evidence of an adversarial attack?
No. Ordinary model errors, natural data drift, sensor noise, and genuinely ambiguous inputs are common and are not evidence of an attack. Calling every mistake "adversarial" overstates the risk and can distract from real vulnerability management.
What is RAG poisoning?
RAG poisoning is planting misleading or malicious content inside a document repository, wiki, or knowledge base that a retrieval-augmented generation system will later fetch and treat as trustworthy context, potentially causing it to give false answers or take unsafe actions.
Key Takeaways
Adversarial machine learning studies deliberate attacks on AI systems, and the defenses against them, across the whole lifecycle, not just at the moment of prediction.
NIST's 2025 taxonomy separates predictive-AI risks (evasion, poisoning, privacy attacks) from generative-AI risks (prompt injection, RAG poisoning, agent misuse).
Attacker knowledge (white-box, grey-box, black-box) and goals (targeted vs. untargeted, integrity vs. availability vs. privacy) define what a specific defense actually protects against.
Adversarial training and certified robustness both help, but neither makes a system universally secure, and both come with real costs.
Generative AI and AI agents expand the attack surface to include everything a model reads and every tool it can call, which is why permissioning, sandboxing, and human oversight matter as much as model-level defenses.
Weak, non-adaptive testing creates a false sense of security; robustness claims are only meaningful with a clearly stated threat model and adaptive evaluation.
Real-world incidents (like the Tay poisoning) and controlled research demonstrations (like physical adversarial patches) teach different lessons and should not be conflated.
Defense in depth, spanning governance, data, model, API, monitoring, and incident response, is the consistent recommendation across NIST, MITRE ATLAS, and OWASP.
Actionable Next Steps
Technical leaders: commission a lightweight threat-modeling exercise for each production AI system this quarter, scoring it by data sensitivity, decision impact, and exposure to untrusted input or retrieved content.
ML teams: add adversarial and out-of-distribution test cases to your existing model-evaluation pipeline, alongside standard accuracy metrics, before the next model release.
Security teams: map your current AI deployments against MITRE ATLAS tactics and the OWASP Top 10 for LLM Applications to identify uncovered categories.
Organisations starting an AML programme: begin with data provenance and API rate limiting, the two controls with the best cost-to-risk-reduction ratio, before investing in expensive adversarial training.
Any team deploying RAG or agents: apply least-privilege permissions and require human approval for any agent action with real-world or financial consequences.
All teams: schedule a recurring (at minimum annual) independent red-team review, since new attacks and new model versions can both silently erode previously validated robustness.
Glossary
Adversarial example: an input crafted to cause a model to make an error.
Adversarial training: training a model on adversarial examples alongside clean data to improve robustness.
Availability attack: an attack aimed at making a system unusable rather than wrong.
Backdoor (trojan): a hidden trigger planted in a model that causes attacker-chosen behaviour only when present.
Black-box attack: an attack where the attacker can only observe a model's outputs, not its internals.
Carlini-Wagner (C&W) attack: an optimization-based attack that minimizes perturbation size while causing misclassification.
Clean-label poisoning: poisoning where the malicious examples keep a correct-looking label.
Concept drift: a natural change in the relationship between inputs and outputs over time; see this site's guide to ,[object Object],.
Data poisoning: deliberately inserting or altering training data to corrupt a model's behaviour.
Evasion attack: an inference-time attack that crafts an input to be misclassified.
FGSM (Fast Gradient Sign Method): a single-step gradient-based method for generating adversarial examples.
Indirect prompt injection: malicious instructions hidden inside content a language model reads, rather than typed directly by the user.
Jailbreaking: attempts to get a model to bypass its safety training; see this site's guide to ,[object Object],.
Membership inference: an attack that tries to determine whether a specific record was part of a model's training data.
Model extraction: querying a model repeatedly to train a substitute model that mimics its behaviour.
Model inversion: reconstructing representative training inputs from a model's outputs or gradients.
NIST AI 100-2e2025: NIST's 2025 taxonomy and terminology report for adversarial machine learning.
Perturbation: the change added to an input to create an adversarial example.
Prompt injection: instructions crafted to override a language model's intended behaviour.
RAG (retrieval-augmented generation) poisoning: planting malicious content in a knowledge base a RAG system retrieves; see this site's guide to ,[object Object],.
Sources & References
NIST (2025, March 24). NIST Trustworthy and Responsible AI Report: Adversarial Machine Learning.
MITRE Corporation. MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems).
OWASP GenAI Security Project (2024, November 17). OWASP Top 10 for LLM Applications 2025.
SecureLayer7 (2026, June 9). OWASP LLM Top 10 (2025): Every Risk Explained.
Molnar, C. Interpretable Machine Learning: Chapter 30, Adversarial Examples.
