What Is a Prompt Injection Attack?
- Jul 29
- 27 min read
Every large language model reads two things at once: the instructions its developer wrote, and whatever text a user or the internet hands it next. It has no reliable way to tell them apart. A prompt injection attack exploits that gap, slipping a hidden instruction into a webpage, email, or document so the AI follows the attacker's command instead of the user's. As AI assistants gain the ability to browse, email, and use tools on our behalf, this single design gap has become the industry's most-discussed security problem, and organisations from OWASP to NIST now treat it as the top risk facing production AI systems.
TL;DR
A prompt injection attack manipulates an AI system by hiding instructions inside content it processes, not just inside what the user types.
OWASP ranks prompt injection as the number-one risk in its Top 10 for LLM Applications 2025.
Direct injection comes from the user's own prompt; indirect injection is smuggled inside webpages, documents, emails, or other data the AI reads later.
Prompt injection is related to, but distinct from, jailbreaking, SQL injection, and other classic security exploits.
No single filter, prompt, or model update fully solves prompt injection; defence needs layered, deterministic controls outside the model.
Risk rises sharply when AI agents can browse, call tools, or take real-world actions on a user's behalf.
What Is a Prompt Injection Attack?
A prompt injection attack is when someone hides malicious instructions inside text an AI system reads, such as a webpage, email, or document, so the AI follows the attacker's commands instead of the user's. It works because most AI models cannot reliably tell trusted instructions apart from untrusted data in their context.
Table of Contents
What Is a Prompt Injection Attack?
A prompt injection attack is a security exploit that manipulates a large language model (LLM) by inserting malicious instructions into the text it processes, causing it to ignore its intended task and follow the attacker's instructions instead. In plain terms: if you can get an AI to read your words, you may be able to get it to obey your words. OWASP formally ranks this as LLM01:2025, the top entry in its Top 10 for LLM Applications, describing it as an input that alters the model's behaviour or output in unintended ways, even when the input isn't visible to a human reader [1].
For a non-technical reader, the simplest description is this: an AI model reads a mix of text, its developer's instructions, the user's question, and often outside content like a webpage or a file, as one continuous stream, without clear walls between things it must obey and things it is simply reading. A foundational research paper on the topic put it plainly: augmenting models with retrieval and other integrations blurs the line between data and instructions [11].
It helps to separate three layers that often get lumped together. The model is the raw large language model itself, the pattern-matching engine that turns text into text. The application is the surrounding software a developer builds, such as a chatbot interface, which decides what the model is shown and what happens with its output. The tools are the external capabilities the application may grant the model: web search, email, code execution, or other APIs. A prompt injection attack targets the model's inability to separate instructions from data, but the actual damage depends almost entirely on what the surrounding application and tools are allowed to do.
This distinction matters because it is easy to overreact. Not every strange or unexpected model response is an attack; models hallucinate and misunderstand ambiguous requests constantly. Prompt injection specifically describes a case where untrusted content is intentionally, or effectively, steering the model away from its intended task by masquerading as an instruction.
How Prompt Injection Attacks Work
Most LLM applications are built with a rough hierarchy in mind: a system prompt or developer instructions sit on top, followed by the user's own prompt, and finally any retrieved or external content the application feeds in. In principle, instructions higher in that hierarchy should always win. In practice, the model receives all of this as one block of tokens, trained to follow instruction-shaped text wherever it appears, with no hard-coded switch that prevents a sentence buried in a webpage from reading, to the model, exactly like a command from the developer [1][2].
This is why application-level trust boundaries matter so much. A trust boundary is the line separating content the developer controls and can vouch for from content that originates outside that control, such as a customer's message or a scraped webpage. Good application design treats everything on the far side of that boundary as unverified, no matter how official it looks.
A simple way to picture the attack path: attacker-controlled content enters the system, the application pulls it into the model's context window, the model produces a response or requests a tool action shaped by the attacker's text, that response reaches downstream application logic, and real-world impact follows if the application acts on it without an independent check.
The critical hinge in that chain is the difference between manipulating output and manipulating action. If a poisoned webpage causes a summarisation tool to describe the page inaccurately, the damage is limited to a wrong answer. If the same page causes an agent with email access to actually send a message or approve a transaction, the output has become an action with real consequences.
Security teams often compare prompt injection to social engineering, and the analogy holds conceptually: in both cases, a message is crafted to make someone, or something, behave against their own interests. OpenAI describes prompt injection explicitly as a type of social engineering attack specific to conversational AI [6]. But it is not identical to human social engineering; a language model has no persistent self-interest to appeal to, and is manipulated because it processes instruction-shaped text probabilistically, without a dependable way to check the credentials of a sender.
Direct vs Indirect Prompt Injection
Direct prompt injection happens when the person interacting with an AI system types the malicious instruction themselves, trying to make the model ignore its system prompt, reveal hidden instructions, or produce content it would normally refuse. Because the attacker and the user are the same person, direct injection is mostly a concern for the operator of an AI system: it can leak internal prompts or produce policy-violating output, but it generally cannot reach beyond what that one user is already allowed to see.
Indirect prompt injection is more consequential, and it is why OWASP and NIST both frame it as the more urgent risk [1][4]. Here, the attacker is a third party with no direct access to the AI system at all. Instead, they plant instructions inside content they expect the AI to read later: a webpage the AI is asked to summarise, an email an AI assistant manages, a resume submitted to a screening tool, a support ticket, a code comment, or a row in a database used for retrieval-augmented generation. When the unsuspecting user, or the system acting on their behalf, asks the AI to process that content, the embedded instruction rides along and is read with the same weight as anything else in the context [11].
Indirect injection can also persist. A poisoned document stored in a company's knowledge base can sit dormant and affect every future AI session that reads it, sometimes described as stored or persistent injection. Attacks can also unfold across multiple steps or sessions, known as cross-context or multi-step injection. And as AI models increasingly accept images, audio, or video, injected instructions have started appearing inside those media too, a category often called multimodal injection.
Indirect injection is harder for an end user to notice for a simple reason: the user never sees the malicious text, only the AI's summary, answer, or action. A hidden instruction on page three of a scanned document, or white text on a white background, is invisible to a human skimming the source, but fully legible to a model reading raw or rendered content.
Aspect | Direct Injection | Indirect Injection |
|---|---|---|
Attacker access | Direct, same person as the user | Remote, no access to the AI system itself |
Entry point | The chat input or prompt field | External content the AI later reads (web, email, files, retrieval) |
Visibility to the user | Usually visible, since the user typed it | Usually invisible, hidden inside processed content |
Typical goal | Bypass safety rules, leak system prompt | Hijack agent actions, steal data, manipulate downstream users |
Prompt Injection vs Jailbreaking and Traditional Injection Attacks
The terms prompt injection and jailbreaking are often used interchangeably, and OWASP notes the two are frequently conflated even though they describe different mechanisms [1]. Jailbreaking is an attempt, usually through direct prompting, to make a model abandon its safety training entirely so it produces content it was built to refuse. Prompt injection is broader: it covers getting the model to follow any unauthorised instruction, which may or may not involve breaking a safety rule. OWASP's framing is useful here: developers can reduce prompt injection risk through system-prompt design and input handling, but reliably preventing jailbreaks generally requires retraining the model's own safety behaviour [1]. A related, narrower goal is system-prompt leakage, where an attacker aims specifically to get the model to reveal its own confidential instructions.
Security teams sometimes describe prompt injection as the SQL injection of AI, a useful first mental model since in both cases untrusted input reaches an interpreter that cannot tell code from data. But the UK's National Cyber Security Centre has explicitly warned that treating the two as equivalent is dangerous [10]. SQL injection has a clean technical fix: parameterised queries create a hard boundary between a query's structure and its data, closing the vulnerability almost completely. Natural language has no equivalent boundary; there is no reliable way to mark a sentence as definitely data, never an instruction, because the model's only real skill is turning any text, marked or not, into more text [10][11]. That is why the NCSC argues prompt injection may be structurally harder to close off for good.
It is also worth distinguishing prompt injection from related concepts. Cross-site scripting and command injection are close cousins conceptually, but target browsers and operating systems rather than language models. Data poisoning corrupts a model during training, permanently altering its behaviour, whereas prompt injection happens at inference time, through the input the model sees on a single occasion. Adversarial examples manipulate output through subtle, mathematically engineered perturbations to input data, particularly in image classifiers, rather than through natural-language instructions.
The Anatomy of a Prompt Injection Attack
Security teams threat-modelling an AI application find it useful to break a scenario into component parts: the attacker's goal, whether data, an action, or a fraudulent output; the entry point where their content gets in; the untrusted content itself; the model context, everything the model actually sees; the target instruction or policy the attacker is trying to override; the data the model can already see; the tools it can call; the authorisation boundary separating what the model may do from what it may not; the model's output or tool request; any application-side validation before acting on it; and the real-world consequence if that validation fails.
Risk climbs sharply when three conditions line up: untrusted input reaches the model, that session has access to sensitive information or privileged capabilities, and there is no deterministic control, something outside the model's own judgement, standing between the model's request and the action happening. Remove any one of the three and the picture changes: a model with no sensitive data or tools can be manipulated but cannot do much damage, and a model with untrusted input and privileges, but a hard permission check and a human approval step in front of every sensitive action, has the injection contained even if it succeeds at the language level. This framing, untrusted input, privilege, and missing deterministic control, is the most useful mental model for deciding where an AI application actually needs defence.
Common Prompt Injection Techniques
Researchers and red teams have catalogued techniques attackers use to make injected instructions land, and OWASP's cheat sheet groups them into recognisable families [2]. None of these should be read as a how-to; they are described here at the level needed to recognise and defend against them.
Instruction override attempts are the most direct family: text explicitly telling the model to ignore its previous instructions. Role or persona manipulation asks the model to pretend to be a different character with fewer restrictions. False authority or urgency framing dresses up the injected text as coming from a developer or administrator. Context manipulation reframes surrounding text so the injected instruction appears to be a natural continuation of the user's own request, making it harder for a filter, or a human, to recognise it as foreign.
Delimiter confusion exploits the markers, such as quotes or code fences, that developers use to separate instructions from data. Encoded, obfuscated, or multilingual instructions hide the payload using common encodings or a language a model's filters were not tuned for. Typographical variations, such as random capitalisation or look-alike characters, rely on the fact that a model can still understand deliberately altered text even when a keyword filter cannot [2]. Hidden text and formatting tricks bury an instruction where a human reviewer will not see it, such as pale text on a matching background or metadata fields that get parsed but never rendered.
Retrieval poisoning targets RAG systems directly, planting an instruction inside a document likely to be pulled into context by a future query. Tool-call manipulation tries to alter the parameters an agent passes to a connected tool. Memory poisoning plants an instruction that survives into future sessions. Multi-turn manipulation spreads a manipulation across several exchanges so no single message looks suspicious, and multimodal injection hides instructions in images, audio, or other non-text input.
A recurring theme: the payload is usually written to look relevant to whatever task the user actually asked for, since on-topic content is far less likely to be questioned. This is also why OWASP is direct that fixed keyword matching is not adequate on its own, since variations and paraphrases can dodge a static list of banned phrases indefinitely [2].
Prompt Injection Examples and Attack Scenarios
The scenarios below are conceptual and synthetic, built to illustrate how prompt injection plays out in real product categories rather than to document any specific incident.
Customer-Support Chatbot
A company deploys an AI chatbot connected to its order database. A customer pastes text into the chat box that reads, in part, like an internal system message granting refund authority. Entry point: the chat input, a direct injection. Goal: an unauthorised refund. Weak control: the bot trusts any instruction-shaped text regardless of who typed it, and can issue refunds without a second check. Impact: financial loss. Mitigation: enforce refund limits in the application layer, not the model's judgement, and treat all user input as data, never as configuration to be obeyed.
AI Assistant Summarising an Untrusted Webpage
A user asks an assistant to summarise an article. The page contains hidden text instructing the model to append a suspicious link to its summary. Entry point: the webpage content. Goal: distribute a malicious link through a trusted assistant's output. Weak control: no separation between the page's content and instructions found within it. Impact: reputational and safety risk. Mitigation: clearly delimit retrieved content, and validate any link the model tries to include before it is displayed.
RAG Assistant Reading a Poisoned Document
An internal knowledge assistant retrieves company documents to answer employee questions. One document has been edited to include an instruction telling the model to reveal salary data whenever it is retrieved. Entry point: the document store used for retrieval. Goal: unauthorised data disclosure. Weak control: the pipeline does not distinguish informational content from instruction-like text, and the model has broad read access. Impact: privacy exposure. Mitigation: apply the same access controls to what a model can retrieve as to a human employee, and treat retrieved passages as untrusted data.
Email Assistant Processing Attacker-Controlled Email Text
An AI assistant with inbox access is asked to handle new emails. An attacker sends a message instructing it to forward all mail from a specific sender to an external address. Entry point: the body of an incoming email. Goal: covert data exfiltration. Weak control: the assistant can create forwarding rules without human confirmation. Impact: ongoing confidentiality loss. Mitigation: require explicit human approval for any rule change or bulk forwarding action.
Coding Assistant Reading Repository Comments or Issue Descriptions
A developer asks an AI coding assistant to fix a bug described in an issue. The issue text contains an instruction telling the assistant to insert a hidden dependency. Entry point: the issue description, treated as trusted input. Goal: supply-chain compromise. Weak control: the assistant can commit changes without a scoped diff review. Impact: potential codebase compromise. Mitigation: sandbox agent-driven code changes, require human review of diffs, and restrict write scope to relevant files.
Browser Agent Encountering Hidden Instructions
A browser-using agent is asked to book a hotel. A page in the booking flow contains an invisible instruction telling the agent to enter payment details on an unrelated third-party form. Entry point: hidden text on a webpage visited mid-task. Goal: payment credential theft. Weak control: the agent acts on any instruction-shaped text it encounters while browsing. Impact: direct financial loss. Mitigation: require explicit confirmation before an agent submits payment information or navigates outside the task's original scope, consistent with the layered defences Anthropic has documented for browser agents [5].
Security Assistant Analysing Attacker-Controlled Log Fields
A security operations assistant summarises log entries for an analyst. An attacker crafts a log field containing an instruction telling the assistant to mark the entry as benign. Entry point: a log field, normally treated as inert data. Goal: evade detection. Weak control: the model reads log fields as potential instruction text too. Impact: a real intrusion goes unflagged. Mitigation: pre-sanitise log fields, and never let a single model call be the sole gate for a security verdict.
Enterprise Agent with File, Calendar, and Communication Access
A broadly scoped assistant agent has access to files, calendar, and messaging. A malicious calendar invite contains an instruction to share a confidential file with an external address listed in the invite's description. Entry point: calendar event metadata. Goal: data exfiltration through an unmonitored channel. Weak control: the agent's permissions span far more than any single task requires. Impact: broad, hard-to-trace data loss. Mitigation: scope agent credentials tightly per task, and require a human-in-the-loop step before external sharing.
Why RAG, AI Agents, Browsers and Tool Use Increase the Risk
Retrieval-augmented generation exists to give a model access to current or proprietary information it was not trained on, but it does so by pulling untrusted external content directly into the model's working context on every query. Relevance is not the same as trustworthiness: a retrieval system finds the passage that best matches a query, not one that judges whether that passage is safe to follow as an instruction. OWASP's own guidance is direct: RAG and fine-tuning do not fully mitigate prompt injection vulnerabilities [1].
Agentic systems raise the stakes further because they have a larger action surface than a simple chatbot. A model that can only produce text can, at worst, say something wrong; a model that can browse, send email, or call other APIs can do something wrong, a fundamentally different risk category. NIST's Center for AI Standards and Innovation ran controlled evaluations of an agent given a legitimate task while encountering data containing a hidden, competing injection task, and found novel attack techniques succeeded at hijacking agents 81% of the time, versus 11% for the strongest previously known baseline attack, when each task was attempted 25 times [4]. Defences tuned only against known, previously published attack patterns can badly underestimate the risk a determined attacker actually poses.
Browser access, email access, code execution, and inter-agent communication each expand what a successful injection can achieve, and excessive permissions compound the danger. An intelligent agent that only needs read access to one calendar but is granted broad account-wide access turns a minor injection into a major one, a version of the classic confused-deputy problem: a program with more privilege than the user directing it can be tricked into misusing that privilege without its own credentials being stolen.
A recurring point across the industry's own research is that the model itself should never be the sole authorisation mechanism for a sensitive action [1][4][5]. Long context windows do not solve this; a bigger context just gives an attacker more room to hide a payload. Fine-tuning raises the bar but, as OWASP notes, does not close the underlying gap [1]. Autonomous retries matter too, since an autonomous agent that keeps trying after an initial refusal faces a higher risk profile than a single-shot interaction, part of why NIST tested multiple attempts per task rather than just one [4]. Tool ecosystems such as the Model Context Protocol deserve the same threat-modelling discipline as any other integration, evaluating what each connected tool can do, not just whether the connection itself is secure [12].
Potential Business and Technical Impact
The consequences of a successful attack range from mildly embarrassing to seriously damaging, and are easiest to reason about through the classic security lens of confidentiality, integrity, and availability. Confidentiality impact includes exposure of sensitive information or leakage of a system prompt. Integrity impact includes manipulated output presented as trustworthy, or unauthorised tool use that changes real records, such as issuing refunds or altering files. Availability impact is less discussed but real: an attack causing an agent to loop or trigger rate limits can disrupt a service for everyone.
Beyond these categories, organisations face fraudulent communications sent under their name, decision manipulation when an injected instruction skews a business recommendation, and security-monitoring evasion. Compliance and privacy exposure follows naturally from unauthorised data disclosure, and reputation damage, operational disruption, and direct incident-response costs can outlast the technical incident itself.
It is worth being precise about what is documented risk and what remains speculative. Controlled evaluations from NIST and Anthropic demonstrate real, measurable attack success rates in test environments [4][5], and OWASP treats the category as an active, top-priority risk based on vulnerabilities found across many production applications [1]. That is different from claiming every deployed agent has already been attacked in the wild; the risk scales directly with how much access and autonomy a given AI system has been granted.
How to Detect Prompt Injection Attempts
Detection works best as a layered process, because no individual technique catches everything. Input and retrieved-content analysis screens text before it reaches the model. Task-relevance checks ask whether any instruction-like text found in retrieved content actually relates to what the user asked for; an instruction to message every contact, buried inside a document to be summarised, fails that test immediately. Prompt-attack classifiers sit alongside the main model; Microsoft's Prompt Shields combine detection with a technique called spotlighting that helps a model distinguish trusted instructions from untrusted content it has been shown [8][9][13].
Pattern and anomaly detection watches for statistically unusual inputs. Tool-call monitoring flags calls that do not match the user's stated task. Permission checks verify, independently of the model, that a requested action targets an allowed resource. Output validation inspects a response before it is finalised, and data-loss prevention watches for sensitive data leaving through an AI-mediated channel. Human review remains valuable because people catch context automated tools miss, and canary techniques, planting a value that should never legitimately appear, offer a lightweight way to catch some exfiltration attempts.
None of this is foolproof. Classifiers produce false positives and false negatives, particularly as attackers adapt their phrasing to evade a known filter. Evaluation needs to account for adaptive, repeated attempts; NIST's own methodology tested each task 25 times precisely because single-attempt testing understated real risk [4]. Detection reduces risk; it does not, by itself, guarantee safety.
How to Prevent Prompt Injection Attacks
There is no single fix for prompt injection, and every credible source in this space says so directly [1][2][10]. Effective defence comes from layering multiple controls so a failure in one does not automatically mean a successful attack.
Start from the assumption that all external content is untrusted. Where possible, separate instructions from data structurally: use the role-based message formats LLM APIs provide, such as system, user, and tool roles, rather than concatenating everything into one string, and mark clear boundaries around external content [2]. Input validation strips known attack patterns before they reach the model, and structured data formats reduce the surface for hidden instructions.
Allowlisted operations restrict what actions an application will ever allow, regardless of what the model asks for. Deterministic policy enforcement outside the model is the single most important principle here: a hard-coded permission check that runs in ordinary application code is far more reliable than asking the model to police itself. Least-privilege access limits what any AI session can see or do to what the current task genuinely requires, and short-lived credentials limit the damage window if an attacker gains some control. Tool-level permissions apply the same logic per tool.
Human approval for sensitive actions, such as sending money or sharing files externally, remains one of the most effective controls, because it introduces a check the model cannot manipulate through language alone. Sandboxing isolates what an agent can touch, and network restrictions stop an agent reaching arbitrary external endpoints even if convinced to try. Data minimisation limits what sensitive information a model ever sees, and output validation checks a response for injected links or scripts before it is rendered.
Security monitoring, red teaming, and regular evaluation keep defences current as attackers adapt, and incident-response preparation ensures a team can act quickly. Defence in depth means an attacker has to defeat several independent layers, not just outsmart a clever system prompt.
It is worth stating what many teams get wrong first: stronger prompting alone is not a complete defence. Asking a model to never follow instructions found in retrieved content measurably helps, but it is a probabilistic nudge, not a guarantee, because the underlying architecture still processes instructions and data as the same kind of text [1][10].
Secure Architecture for LLM Applications and AI Agents
A secure LLM application separates the parts that must be trusted from the parts that never can be. The trusted application policy is the code a developer writes and controls directly. The untrusted content boundary marks everything arriving from outside that trust: user messages, retrieved documents, tool results, browsed pages. The retrieval layer fetches external content but hands it to the model clearly labelled as data, never as instruction. The model reasons over all of this and proposes a response or action.
The policy-enforcement point is where a proposed action meets a hard, code-level check before anything happens, the architectural heart of a defensible system. A tool broker mediates every tool call with scoped permissions rather than direct access. An identity and authorisation service confirms, independently of the model's own claims, what the session is actually allowed to do. An approval workflow inserts a human checkpoint for actions above a risk threshold. A logging and audit system records what happened. An output validator checks the final response for injected content before it reaches the user. A sandbox contains what agent-driven code execution can touch, and incident-response controls exist and are tested before they are ever needed.
The core principle: the model may propose an action, but deterministic code must validate it; authorisation is checked independently of anything the model asserts; sensitive actions require explicit human approval; every tool result coming back is treated as untrusted data going forward; and the model receives only the minimum data and capability the task actually needs.
A short comparison makes the difference concrete:
// UNSAFE: model output executed directly
action = model.decide(userInput, retrievedContent)
tool.execute(action) // no independent check
// SAFER: policy enforcement between model and action
proposal = model.decide(userInput, retrievedContent)
if not policy.isAllowed(proposal, session.permissions):
reject(proposal)
elif proposal.riskLevel == "HIGH":
requestHumanApproval(proposal)
else:
tool.execute(policy.scope(proposal))
log(proposal, decision)The unsafe version trusts the model's decision completely. The safer version treats the output as a proposal, checks it against a policy the model cannot rewrite, escalates high-risk actions to a person, and records what happened either way.
Testing, Red Teaming and Incident Response
Testing starts with threat modelling: mapping out exactly what data an application can see, what tools it can call, and what an attacker would want to achieve through it. Static test sets catch obvious gaps quickly but say little about attacks nobody has published yet. Adaptive attacks and repeated-attempt testing give a far more realistic picture; NIST's evaluation work found testing each scenario 25 times surfaced meaningfully higher hijacking rates than a single-attempt test would have suggested [4].
A thorough programme covers direct and indirect injection, multimodal inputs, RAG pipelines, and tool-use boundaries, probing what actually happens if an injection is not recognised. Cross-session tests check whether an instruction can persist into later, unrelated interactions. Evaluation should be framed around business impact, and any serious programme grapples with the trade-off between attack-success rate and utility, since a system that refuses legitimate requests constantly is not a usable win either. Logging lets a team confirm a fix actually worked, third-party testing catches blind spots an internal team develops, and regression tests ensure a defence that worked last quarter still works after an update.
When an attack succeeds despite these precautions, incident response follows familiar security discipline: kill switches to pause a feature, credential revocation, containment, investigation, and a considered decision on user notification based on what was actually affected. Lessons learned should feed back into the test suite. Passing a fixed benchmark does not prove immunity; it proves the system resisted the attacks that were tried. Given how quickly attackers adapt, ongoing evaluation is a permanent requirement, not a one-time certification [1][4].
Prompt Injection Prevention Checklist
Data
Treat every external document, webpage, email, and retrieved passage as untrusted, regardless of its apparent source.
Apply the same access controls to what a model can retrieve as you would to a human with the same task.
Minimise the sensitive data included in any prompt or context window.
Prompts
Use role-based message structures, such as system, user, and tool, instead of concatenating text into one string.
Clearly delimit or label any external content passed to the model as data, not instruction.
Do not rely on prompt wording alone as your only defence.
Identity and Permissions
Grant each AI session the minimum access it needs for its current task, and nothing more.
Use short-lived, narrowly scoped credentials wherever the platform supports them.
Apply distinct permission scopes per tool, not one broad grant.
Tools and Actions
Route every proposed model action through a deterministic, code-level policy check.
Require explicit human approval for financially significant, destructive, or externally visible actions.
Sandbox any agent-driven code execution and restrict network destinations.
Monitoring
Log tool calls, retrieved content, and model outputs in enough detail to investigate later.
Watch for output that does not match the user's stated task.
Apply data-loss-prevention screening to anything leaving the system.
Testing
Run adaptive, repeated-attempt red-team tests, not just static payload lists.
Test direct injection, indirect injection, and multimodal inputs where relevant.
Add every discovered gap to a permanent regression suite.
Incident Response
Maintain a tested kill switch or pause mechanism for agentic features.
Prepare credential-revocation and containment procedures in advance.
Define notification criteria for confirmed data or action impact before an incident happens.
The Future of Prompt Injection Security
Several trends point toward this problem becoming more, not less, important. Growing agent autonomy means AI systems are trusted with longer chains of action and less human oversight per step. Multimodal systems widen the attack surface into images, audio, and video, where injected instructions are even less visible to a human reviewer than text is. Longer context windows give attackers more space to hide a payload, and cross-agent communication creates new paths for an injected instruction to travel between systems that were each individually tested but never tested together.
Attackers are adaptive by nature; NIST's own finding that novel methods dramatically outperformed known baselines is itself evidence the offensive side of this problem is advancing [4]. On the defensive side, providers are investing in training models to resist injected instructions, and Anthropic's reported reduction in browser-agent attack success, while explicitly not claiming the problem is solved, shows meaningful progress is possible [5].
What is unlikely to change is the need for ongoing evaluation. Prompt injection remains an active research area because it is rooted in how language models fundamentally work, not a bug a single patch can close. Organisations building or buying AI agents should expect this to be a permanent line item in their AI security programme.
FAQ
What is a prompt injection attack in simple terms?
It is when someone hides instructions inside text an AI reads, such as a webpage or file, so the AI follows the hidden command instead of doing what its user wanted. It works because most AI models process instructions and ordinary content as the same kind of text [1].
What is an example of prompt injection?
A common example is indirect injection: an attacker hides an instruction inside a webpage an AI assistant is asked to summarise. The user only asked for a summary, but the hidden text tells the model to also insert a suspicious link. The user never sees the instruction, only the AI's altered output.
What is the difference between direct and indirect prompt injection?
Direct injection comes from the person using the AI system, typed straight into the prompt. Indirect injection comes from a third party with no direct access, planted inside content such as a document or webpage the AI reads later. Indirect injection is generally the more serious risk because the end user cannot see it happening.
Is prompt injection the same as jailbreaking?
No. Jailbreaking means getting a model to abandon its safety training and produce refused content. Prompt injection is broader; it covers any case where unauthorised instructions steer a model's behaviour, which may or may not involve breaking a safety rule [1].
Can prompt injection steal data?
Yes, if the AI system has access to sensitive data and no independent check on what it shares. An injected instruction can direct a model to disclose information from its context, retrieved documents, or connected tools.
Can prompt injection affect RAG systems?
Yes. Retrieval-augmented generation pulls external documents into a model's context, and an attacker can plant an instruction inside any document that might get retrieved. OWASP notes RAG does not fully solve prompt injection, since retrieval checks relevance, not trustworthiness [1].
Are AI agents more vulnerable to prompt injection?
Agents face higher stakes rather than higher vulnerability at the language level. Because agents can browse, call tools, and act, a successful injection can cause tangible harm. NIST found novel attack techniques hijacked agents at a much higher rate than previously known methods [4].
Can system prompts prevent prompt injection?
A well-written system prompt reduces risk by treating retrieved content as data rather than commands, but cannot fully prevent injection alone, since the model still processes both as the same kind of text [1][10].
Can input filtering stop prompt injection?
It helps catch obvious patterns, but fixed keyword matching is easy to bypass through rewording, encoding, or unusual spacing. OWASP treats filtering as one layer among several, not a standalone fix [2].
How can developers test for prompt injection?
Effective testing combines static payload tests with adaptive, repeated-attempt red teaming across direct injection, indirect injection, and tool-use pipelines. NIST's approach showed single-attempt tests understate real risk [4].
What should a company do after detecting an attack?
Standard incident-response steps apply: contain the immediate risk, revoke exposed credentials, investigate scope and root cause, and decide on user notification based on what was actually affected. The exploited gap should become a permanent test.
Is prompt injection completely preventable?
No credible source claims that. OWASP, NIST, and multiple AI providers describe it as a risk reduced through layered defence, not eliminated by any single fix [1][4][5].
Why do RAG and fine-tuning not fully solve prompt injection?
Both improve relevance or accuracy, but neither changes the fact that the model still processes instructions and untrusted data as the same kind of text. OWASP states RAG and fine-tuning do not fully mitigate prompt injection [1].
What is the confused-deputy problem in AI security?
It describes a program with more privilege than the person directing it, which can be tricked into misusing that privilege without the attacker stealing its credentials. AI agents with broad tool access are a modern example.
Does a strong AI model guarantee safety from prompt injection?
No single model guarantees safety alone. Even providers reporting measurable improvements are explicit that a low attack-success rate still represents meaningful risk, particularly for agents handling sensitive data or actions [5].
Key Takeaways
Prompt injection exploits the fact that most AI models cannot reliably separate trusted instructions from untrusted data in their context.
OWASP ranks prompt injection as the top risk facing LLM applications, ahead of every other category in its 2025 Top 10 [1].
Direct injection comes from the user; indirect injection is hidden in content the AI reads later, and is generally the more dangerous form.
Prompt injection is related to, but distinct from, jailbreaking, SQL injection, and data poisoning.
Risk rises sharply with agent autonomy; the more an AI system can browse, call tools, or act, the more a successful injection can achieve.
No single defence works alone; effective security relies on layered, deterministic controls outside the model, not stronger prompting alone.
Ongoing testing, monitoring, and incident-response readiness are permanent requirements, not one-time fixes.
Actionable Next Steps
Inventory every AI feature that processes external content or has tool access, and note what data and permissions each one actually has.
Apply least-privilege access to every AI session, scoping credentials narrowly and using short-lived tokens where possible.
Move sensitive-action approval outside the model: require deterministic policy checks and human sign-off for anything financially significant or destructive.
Treat all retrieved, browsed, or uploaded content as untrusted data in your prompt design, never as instructions.
Build an adaptive, repeated-attempt red-team test suite covering direct injection, indirect injection, and your tool-use pipelines.
Establish incident-response procedures, such as kill switches and credential revocation, before you need them.
Glossary
AI agent: An AI system that can plan, use tools, and take actions toward a goal. See What Are AI Agents.
Allowlist: A list of explicitly permitted actions or values; anything not listed is denied by default.
Data exfiltration: The unauthorised transfer of data out of a system.
Direct prompt injection: Malicious instructions submitted straight into an AI system's prompt by the person using it.
Human-in-the-loop: A design requiring a person to review or approve an AI action before it takes effect.
Indirect prompt injection: Malicious instructions hidden inside external content that an AI system processes later, without the end user seeing them.
Jailbreak: An attempt to make an AI model abandon its safety training. See AI jailbreaking.
Large language model (LLM): An AI model trained on text to understand and generate natural language.
Least privilege: Granting a system only the minimum access needed for a specific task.
Multimodal model: An AI model that can process more than one input type, such as text, images, and audio.
Prompt: The text or instructions given to an AI model to generate a response.
Prompt injection: An attack manipulating an AI system's behaviour by embedding unauthorised instructions within content it processes.
Retrieval-augmented generation (RAG): A technique supplying an AI model with external documents at query time.
Red teaming: Deliberately testing a system's defences by simulating realistic attacks.
Sandbox: An isolated environment where an agent's actions are contained.
System prompt: The instructions a developer gives an AI model defining its role and behaviour.
Tool calling: A capability letting an AI model invoke external functions or APIs.
Trust boundary: The line separating content the developer controls from anything outside that control.
Untrusted content: Any data an AI system processes that did not originate from a verified source.
Validation: Checking that an input or proposed action meets safe criteria before it is accepted.
Sources & References
[1] OWASP GenAI Security Project. "LLM01:2025 Prompt Injection." OWASP, 2025. https://genai.owasp.org/llmrisk/llm01-prompt-injection/. Accessed July 29, 2026.
[2] OWASP Cheat Sheet Series. "LLM Prompt Injection Prevention Cheat Sheet." OWASP, 2025. https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html. Accessed July 29, 2026.
[3] OWASP Cheat Sheet Series. "AI Agent Security Cheat Sheet." OWASP, 2025. https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html. Accessed July 29, 2026.
[4] National Institute of Standards and Technology, Center for AI Standards and Innovation. "Technical Blog: Strengthening AI Agent Hijacking Evaluations." NIST, released January 17, 2025, updated December 19, 2025. https://www.nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations. Accessed July 29, 2026.
[5] Anthropic. "Mitigating the Risk of Prompt Injections in Browser Use." Anthropic, November 24, 2025. https://www.anthropic.com/research/prompt-injection-defenses. Accessed July 29, 2026.
[6] OpenAI. "Understanding Prompt Injections: A Frontier Security Challenge." OpenAI, November 7, 2025. https://openai.com/index/prompt-injections/. Accessed July 29, 2026.
[7] OpenAI. "Understanding Prompt Injections." OpenAI Safety. https://openai.com/safety/prompt-injections/. Accessed July 29, 2026.
[8] Microsoft Azure. "Enhance AI Security with Azure Prompt Shields and Azure AI Content Safety." Microsoft Azure Blog, June 5, 2025. https://azure.microsoft.com/en-us/blog/enhance-ai-security-with-azure-prompt-shields-and-azure-ai-content-safety/. Accessed July 29, 2026.
[9] Microsoft. "Azure AI Announces Prompt Shields for Jailbreak and Indirect Prompt Injection Attacks." Microsoft Tech Community, March 28, 2024. https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/azure-ai-announces-prompt-shields-for-jailbreak-and-indirect-prompt-injection-at/4099140. Accessed July 29, 2026.
[10] National Cyber Security Centre (UK). "Prompt Injection Is Not SQL Injection (It May Be Worse)." NCSC, December 8, 2025. https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection. Accessed July 29, 2026.
[11] Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection." arXiv:2302.12173, May 5, 2023. https://arxiv.org/abs/2302.12173. Accessed July 29, 2026.
[12] Microsoft for Developers. "Protecting Against Indirect Prompt Injection Attacks in MCP." Microsoft, April 28, 2025. https://developer.microsoft.com/blog/protecting-against-indirect-injection-attacks-mcp/. Accessed July 29, 2026.
[13] Hines, K., Lopez, G., Hall, M., Zarfati, F., Zunger, Y., and Kiciman, E. "Defending Against Indirect Prompt Injection Attacks With Spotlighting." arXiv:2403.14720, 2024. https://arxiv.org/abs/2403.14720. Accessed July 29, 2026.
[14] Microsoft Learn. "Shield Prompt, REST API, Azure AI Content Safety." Microsoft, updated September 1, 2024. https://learn.microsoft.com/en-us/rest/api/contentsafety/text-operations/shield-prompt?view=rest-contentsafety-2024-09-01. Accessed July 29, 2026.
