What Is AI Model Security?
- 1 day ago
- 29 min read
A model can be 99% accurate and still be completely insecure. In June 2025, security researchers at Aim Security disclosed EchoLeak (CVE-2025-32711), a zero-click flaw in Microsoft 365 Copilot that let a single crafted email pull sensitive files out of a victim's tenant with no click, no download, and no warning sign. Nothing was wrong with the model's predictions. The weakness was in how the surrounding system handled untrusted content, identity, and data flow — which is exactly what AI model security is built to address.
TL;DR
• AI model security protects an AI model and the full system around it — data, weights, APIs, tools, and identities — from theft, manipulation, and misuse, not just from bad predictions.
• The core threats fall into a handful of buckets: data poisoning, model theft, prompt injection, jailbreaking, and sensitive-information disclosure.
• Foundation models, LLMs, RAG pipelines, and autonomous agents each add their own attack surface on top of classic ML risk.
• No single control — not a system prompt, not a filter, not one red-team exercise — secures an AI system. Defense in depth across data, identity, infrastructure, and human oversight is required.
• Frameworks such as the NIST AI RMF, the OWASP GenAI Top 10 lists, and MITRE ATLAS give teams a shared vocabulary and a starting checklist, but none of them substitute for an actual inventory, threat model, and monitoring programme.
• The fastest first step for most organisations is a plain inventory: which models, whose data, which tools, which identities, and who owns each one.
What Is AI Model Security?
AI model security is the practice of protecting an AI model and its surrounding system — training data, weights, APIs, tools, and infrastructure — from theft, manipulation, and misuse. It covers confidentiality, integrity, and availability across the entire AI lifecycle, not just the accuracy of the model's predictions.
Table of Contents
AI Model Security: A Plain-English Definition
In one sentence: AI model security is the set of practices that keep an AI model, its data, and everything connected to it safe from theft, tampering, and abuse.
A fuller definition treats the model as two things at once. It is a software artefact — a file, a set of weights, code that runs on a server — and it is also a learned statistical system that behaves differently depending on the data it was trained on and the inputs it receives at run time. Traditional cybersecurity practice handles the first part well: patch servers, rotate credentials, restrict network access. It has almost nothing to say about the second part, where an attacker never touches a server yet still manipulates what the model does by controlling the data it learns from or the text it reads.
That is why application security remains necessary but not sufficient. A perfectly hardened API in front of a poisoned model still produces manipulated output. A model with pristine training data but an open inference endpoint can still be copied wholesale through repeated queries. Both halves have to be covered.
A useful way to separate the two levels: model security focuses on the model artefact itself — its weights, training data, and learned behaviour. AI system security covers everything the model touches: the application, the retrieval layer, the tools an agent can call, the identities involved, and the infrastructure underneath. Protecting only the weights and ignoring the rest is like locking a vault door while leaving every window in the building open.
Security teams usually describe what they are protecting using three classic properties, plus a few AI-specific ones:
• Confidentiality — keeping training data, prompts, proprietary weights, and user inputs from leaking to people who should not see them.
• Integrity — making sure the model's training data, weights, and outputs have not been tampered with.
• Availability — keeping the model responsive and affordable to run, rather than knocked offline or bankrupted by abuse.
• Authenticity, robustness, resilience, controllability, and accountability — whether outputs can be traced to a legitimate source, whether the system holds up under adversarial input, how fast it recovers, whether humans can steer it, and whether actions can be audited after the fact.
A simple analogy: think of an AI model like a highly capable new employee who has read a huge amount of material and now answers questions and takes actions on the company's behalf. You would not let that employee near sensitive systems without a background check, defined permissions, and a supervisor reviewing their major decisions. The analogy breaks down in one important place — a human employee has judgement and self-preservation instincts that no current AI system possesses, so oversight has to be built externally, not assumed to emerge from the model itself.
Why AI Model Security Matters
The consequences of weak AI model security are concrete, not abstract. In April 2023, Samsung engineers pasted proprietary semiconductor source code and confidential meeting notes into ChatGPT to help debug and summarise their work. Within twenty days the company logged three separate incidents and banned employee use of external generative AI tools soon after (Bloomberg, May 2, 2023). No attacker was involved — the exposure came from ordinary employees trying to do their jobs faster.
That distinction matters throughout this article. Not every bad outcome from an AI system is an attack. It helps to separate:
• Accidental failure — a model gives a wrong answer because of noisy data or an edge case, with no adversary involved.
• Misuse — a legitimate user applies the system in a way it was not designed for, such as pasting secrets into a public chatbot.
• Abuse — someone uses the system's intended features to cause harm, such as generating spam at scale.
• Adversarial attack — an attacker deliberately manipulates inputs, data, or the model itself to force an unintended outcome.
• Security vulnerability — a flaw in the system — code, configuration, or design — that an attacker could exploit, whether or not it has been exploited yet.
• Safety failure — the model does what it was asked to do, but the outcome is still harmful because of poor alignment or missing guardrails.
Once that vocabulary is in place, the business consequences of getting AI model security wrong become easier to categorise: exposure of sensitive information, theft of costly-to-train intellectual property, manipulated or unreliable decisions in high-stakes systems, fraud through impersonation or unauthorised transactions, safety harm in domains like healthcare or transport, regulatory exposure under frameworks such as the EU AI Act, contractual liability toward customers and partners, operational disruption when a model or its API is knocked offline, unexpectedly large inference or cloud costs from abuse, and the reputational damage that follows any of the above becoming public.
What Does AI Model Security Protect?
Securing “the model” is a misleading shorthand. Effective programmes draw the security boundary around the entire data and action flow, not just the file containing weights. That includes:
• Training and evaluation data, data pipelines, labels, and metadata — the raw material a model learns from.
• Model architecture, weights, checkpoints, and fine-tuning artefacts — the trained system itself, including intermediate versions.
• Embeddings and vector databases — numerical representations of content used for search and retrieval.
• Prompts, system instructions, and RAG sources — the context a model reasons over at run time.
• Memory and context stores — anything an agent or chat session remembers between turns.
• Model registries, model files, and serialisation formats — where trained models are catalogued and stored before deployment.
• APIs, inference endpoints, and orchestration frameworks — the software that lets applications call the model.
• Agent tools, plugins, and connectors — the actions an autonomous agent can actually take, from sending email to running code.
• Identities, tokens, and secrets — the credentials that determine what a model or agent is allowed to do.
• Cloud infrastructure, GPUs, and deployment environments — the physical and virtual machinery underneath everything else.
• Logs and monitoring data — the evidence needed to detect and investigate incidents.
• Human approval processes and third-party models or services — the people and vendors whose decisions and code the system depends on.
A security boundary drawn only around the inference endpoint misses most of this list. If an attacker cannot reach the API but can poison a public dataset the model was fine-tuned on, or compromise a vector database feeding its retrieval layer, the endpoint was never the weak point to begin with.
AI Model Security vs. AI Safety, Privacy, and Cybersecurity
These adjacent fields are often used interchangeably, which causes real confusion in how organisations staff and fund their work.
• AI model security — protecting the model and its data from theft, tampering, and adversarial manipulation.
• AI application security — securing the software layer around the model: APIs, authentication, and the usual web-application concerns.
• AI safety — making sure a model's behaviour aligns with intended goals and does not cause harm, even with no attacker present.
• Responsible AI and AI governance — organisational policy, oversight, and accountability for how AI is built and used.
• Data protection and privacy — legal and technical controls over personal data, which AI systems can violate in new ways such as memorisation.
• Model robustness — how well a model performs under noisy, shifted, or unusual input, whether adversarial or natural.
• Traditional cybersecurity — the established discipline of protecting networks, systems, and data, which AI systems still depend on completely.
• Using AI to perform cybersecurity work — applying AI tools to threat detection or analysis, which is a use case for AI, not a description of securing AI itself.
The overlaps are real: a model that leaks training data is simultaneously a security failure and a privacy failure. A jailbroken model that gives dangerous instructions is both a security and a safety event. That is exactly why coordination across security, privacy, legal, and product teams works better than treating each field as its own silo with no shared escalation path.
A Compact Comparison
Domain — Primary Question — Typical Owner
• AI model security — Can this be stolen, poisoned, or manipulated? — Security engineering
• AI safety — Does this behave as intended, even without an attacker? — ML / research teams
• AI governance — Who is accountable, and under what policy? — Risk, legal, executive leadership
• Privacy — Is personal data protected and used lawfully? — Privacy / legal
• Traditional cybersecurity — Is the underlying infrastructure hardened? — IT / security operations
Who Attacks AI Models—and Why?
Threat actors targeting AI systems are more varied than the popular image of a lone hacker. Realistic adversaries include opportunistic external attackers scanning for exposed endpoints, organised cybercriminals seeking financial gain, competitors interested in stealing a costly-to-train model, malicious or careless insiders and contractors, compromised suppliers whose software or data feeds into the system, hostile users probing a public chatbot, authorised researchers conducting sanctioned red-team testing, and attackers who never touch the model directly but instead manipulate the public data or web content a model later ingests.
Their objectives vary just as widely: financial gain, data theft, intellectual-property theft, sabotage, fraud, espionage, censorship or narrative manipulation, resource theft through cost-exhaustion abuse, service disruption, bypassing content or policy controls, taking unauthorised actions through a connected agent, or establishing persistence inside an agent's memory or connected systems.
Access level shapes what an attacker can realistically do. Black-box access means the attacker can only send inputs and observe outputs — the position of most public chatbot users. Grey-box access adds partial knowledge, such as documentation or a similar open model. White-box access means the attacker has the model weights or training code directly, turning many theoretical attacks into practical ones. Most published research assumes white-box or grey-box access; most real-world abuse happens in a black-box setting, which is why input-level defences matter so much in production.
The AI Model Security Lifecycle
Security work needs to happen at every stage of an AI system's life, not just once before launch.
Planning and Threat Modelling
Assets: requirements and data-sensitivity classification. Threats: unclear ownership and undefined trust boundaries. Controls: a written threat model before data collection starts. Owner: product and security jointly.
Data Acquisition and Preparation
Assets: raw and labelled training data. Threats: data poisoning through compromised sources. Controls: source vetting and dataset versioning. Owner: data engineering.
Training and Fine-Tuning
Assets: model weights and compute infrastructure. Threats: backdoors from poisoned fine-tuning data. Controls: isolated training environments and checkpoint signing. Owner: ML engineering.
Evaluation and Red Teaming
Assets: evaluation datasets and benchmark results. Threats: test sets that miss adversarial conditions. Controls: adversarial test suites separate from quality benchmarks. Owner: ML engineering and security.
Packaging and Supply-Chain Transfer
Assets: model files and dependencies. Threats: insecure deserialisation and compromised packages. Controls: scanning model files before they move downstream. Owner: security engineering.
Deployment
Assets: inference infrastructure and secrets. Threats: overly broad access and weak isolation. Controls: least-privilege identities and network segmentation. Owner: platform and security engineering.
Inference and Application Use
Assets: prompts, retrieved content, tool calls. Threats: prompt injection and information disclosure. Controls: input handling, output validation, tool permission limits. Owner: application and security teams.
Monitoring and Maintenance
Assets: logs, telemetry, spend data. Threats: undetected drift and quiet policy-bypass attempts. Controls: continuous evaluation and anomaly alerting. Owner: security operations.
Model Updates
Assets: new weights, changed prompts, updated tools. Threats: a control valid on one version silently failing on the next. Controls: re-testing after every material change. Owner: ML engineering and security.
Decommissioning and Data Disposal
Assets: retired models and old credentials. Threats: forgotten endpoints left accessible. Controls: decommissioning checklists and credential revocation. Owner: platform engineering.
Major AI Model Attacks and Vulnerabilities
The categories below are described at the level needed for defenders to recognise and mitigate them — not as exploit instructions.
Data and Model Poisoning
Data poisoning manipulates training data so the model learns a flawed or backdoored pattern; model poisoning manipulates the training process or weights directly. An attacker needs some influence over a trusted data source. The result ranges from degraded accuracy to a hidden backdoor. Systems fine-tuned on scraped or community-submitted data are most exposed. Main controls: source vetting and anomaly detection on training data.
Backdoors and Trojans
A backdoored model behaves normally until it sees a specific trigger, at which point it produces attacker-chosen output. This requires influence over training or distribution of a pre-trained model containing the trigger. Unverified model reuse is the most exposed path. Main controls: provenance verification and testing against known trigger patterns.
Evasion Attacks and Adversarial Examples
Evasion attacks add small, often imperceptible perturbations to an input so a classifier misreads it — a stop sign misread as a speed-limit sign is the textbook example documented in NIST's adversarial machine learning taxonomy. This needs only query access, sometimes just black-box access. Vision and fraud-detection systems are most exposed. Main controls: adversarial training and input sanitisation.
Prompt Injection and Indirect Prompt Injection
Direct prompt injection happens when a user types instructions designed to override a system's rules. Indirect prompt injection embeds those instructions inside content the model later ingests — a web page, an email, a document — with no need for the attacker to interact with the system at all. It is important to be precise here: prompt injection is not the same as SQL injection. SQL injection exploits a structured query language that can be sanitised with strict rules; natural language cannot be sanitised the same way, because the model is designed to follow instructions written in ordinary language, including ones it should not trust. This is what made EchoLeak and the August 2024 Slack AI data-exfiltration disclosure possible — both exploited a model's inability to separate trusted instructions from untrusted retrieved content. Systems that automatically read external content — email, web pages, shared documents — are most exposed. Main controls: treating all retrieved content as untrusted, isolating instructions from data, and restricting what a model's output can trigger.
Jailbreaking
Jailbreaking gets a model to ignore its safety training and produce content it was designed to refuse, often through role-play framing. This needs only conversational access. Public chat interfaces are most exposed. Main controls: layered filtering and continuous adversarial testing, since alignment alone does not guarantee resistance.
Sensitive-Information Disclosure and Training-Data Leakage
A model can reveal information it memorised during training, or repeat sensitive content fed to it earlier in the same session. This needs only the right prompts. Models fine-tuned on proprietary or regulated data are most exposed. Main controls: data minimisation and output filtering.
Membership Inference and Model Inversion
Membership inference determines whether a record was part of a model's training data; model inversion reconstructs approximate training examples from outputs. Both usually need white-box or grey-box access and repeated querying. Models trained on small or sensitive datasets are most exposed. Main controls: differential privacy and query-volume limits.
Model Extraction and Theft
Extraction rebuilds a functional copy of a model by querying it repeatedly and training a substitute; theft copies the weights file directly. Extraction needs sustained API access; theft needs a compromised credential. Publicly exposed APIs with unlimited queries are most exposed. Main controls: rate limiting and strict access control on weight storage.
Supply-Chain Compromise and Insecure Model Files
Many ML file formats allow code execution during loading if deserialisation is unrestricted — loading an untrusted model file can be equivalent to running unknown code. Teams pulling pre-trained models from public hubs without verification are most exposed. Main controls: restricted-deserialisation formats and scanning files before loading.
Denial of Service, Unbounded Consumption, and API Abuse
Attackers can send resource-intensive queries to exhaust compute, or inflate a victim's cloud bill — sometimes called “denial of wallet.” Pay-per-use APIs without quotas are most exposed. Main controls: token, request, and spending limits enforced outside the model.
Excessive Permissions, Weak Secrets Management, and Insecure Logging
An agent with broader access than its task requires turns a contained prompt injection into a system-wide breach; secrets embedded in prompts or logs can leak the same way. Agentic systems with standing credentials are most exposed. Main controls: least privilege, short-lived credentials, and log redaction.
Shadow AI and Unapproved Model Use
Employees adopting AI tools outside official channels — exactly what happened at Samsung in 2023 — creates exposure with no attacker required. Any organisation without an approved-tool list is exposed. Main controls: an approved-tool policy and staff training.
Security Risks in LLMs, RAG, Multimodal AI, and Agents
Large Language Models
The central weakness across all LLM deployments is instruction-data confusion: the model cannot fully separate developer instructions from user input and retrieved content, which is what makes both prompt injection and system-prompt leakage possible.
Retrieval-Augmented Generation
In a RAG system, retrieval poisoning — planting malicious content where the retriever will find it — and cross-tenant retrieval — one customer's assistant surfacing another customer's documents — are the two failures worth designing against from day one.
Embeddings and Vector Databases
A vector database that mixes access levels without per-document permission checks effectively bypasses whatever access control exists at the application layer, since a retriever will happily pull in anything it can reach.
Multimodal Models
Instructions hidden inside images, audio, or video can reach a multimodal model the same way hidden text reaches a language model, and are often harder for a human reviewer to spot during testing.
AI Agents
An agent adds goal hijacking, tool misuse, and excessive agency to the list — the difference between a chatbot producing a bad sentence and an agent taking a bad, sometimes irreversible, action.
Multi-Agent Systems
When agents communicate with each other, a single compromised or careless agent can cascade its failure across the whole system, particularly if agents trust each other's outputs the way they trust a human's.
Locally Run and Device Models
Models running on end-user devices face physical extraction risk and typically lack the centralized monitoring that a hosted API has by default.
Third-Party Hosted Models
Relying on an external provider means inheriting their security posture, uptime, and policy changes — a dependency that needs its own contractual and technical safeguards.
The point that ties all of this together: model-level safeguards alone cannot secure an agent that has broad permissions and unsafe tools. A perfectly aligned model connected to an email-sending tool with no human checkpoint is still one crafted document away from sending data to an attacker, exactly as EchoLeak demonstrated in a production system used by more than ten thousand organisations.
Realistic AI Model Attack Scenarios
The following scenarios are hypothetical illustrations built from documented attack patterns, not accounts of specific unnamed incidents.
Scenario: A Poisoned Fine-Tuning Pipeline
Initial weakness: a fine-tuning job pulls community-submitted examples with no vetting. Attack path: an attacker submits examples containing a rare trigger phrase paired with a harmful response. Potential impact: the model behaves normally except when it sees the trigger, at which point it produces attacker-controlled output. Detection opportunity: behavioural testing against a library of known trigger patterns before release. Preventive controls: source vetting and a held-out clean validation set that flags behavioural drift.
Scenario: An Indirect Prompt Injection Through a Shared Document
Initial weakness: an assistant automatically summarises any document a user uploads or shares. Attack path: an attacker embeds hidden instructions in a document's white text or metadata, and the assistant follows them as if they came from the user. Potential impact: exfiltration of other content the assistant can access in that session. Detection opportunity: monitoring for unusual outbound requests immediately following document ingestion. Preventive controls: treating all document content as untrusted data, never as instructions.
Scenario: A Multi-Tenant RAG Data Leak
Initial weakness: a shared vector database lacks per-tenant access filters at the retrieval layer. Attack path: a query from one tenant's session retrieves and surfaces another tenant's indexed documents. Potential impact: confidential cross-customer data disclosure. Detection opportunity: retrieval-authorization failure logging. Preventive controls: document-level access control enforced before retrieval, not after generation.
Scenario: An Overprivileged Support Agent
Initial weakness: a customer-support agent has standing write access to the billing system to speed up refunds. Attack path: a crafted support ticket manipulates the agent into issuing an unauthorised refund or account change. Potential impact: financial loss and account compromise. Detection opportunity: anomalous refund-pattern alerts. Preventive controls: requiring human approval for any financially significant action.
Scenario: A Resource-Exhaustion Attack Against an Inference API
Initial weakness: a public inference endpoint has no per-user rate limit. Attack path: an attacker sends a high volume of expensive queries, inflating cost and degrading service for legitimate users. Potential impact: denial of service and an unexpected cloud bill. Detection opportunity: real-time cost and request-rate monitoring. Preventive controls: token, request, and spending caps enforced at the gateway.
How to Secure AI Models with Defence in Depth
No single filter, prompt tweak, or alignment technique secures a production AI system on its own, because each control addresses one layer while attackers look for whichever layer was left uncovered. Effective programmes stack preventive, detective, responsive, and recovery controls across every layer described earlier in the lifecycle.
Preventive Controls
• Maintain an inventory of models, versions, owners, data sources, deployments, and connected tools.
• Classify models by data sensitivity, autonomy, exposure, and potential impact.
• Sign and verify model artefacts, and track lineage and provenance from data to deployment.
• Scan model files and dependencies before they enter a pipeline.
• Separate development, evaluation, and production environments.
• Use narrowly scoped identities and keep secrets out of prompts and model-accessible context.
• Apply document-level access control before RAG retrieval, and treat all retrieved content as untrusted.
• Limit tool access and tool parameters for any agent, and isolate code execution in a sandbox.
• Set token, request, time, and spending limits on every inference endpoint.
Detective and Responsive Controls
• Validate model outputs before they reach interpreters, databases, browsers, shells, APIs, or physical systems.
• Require human confirmation for high-impact or irreversible actions.
• Log prompts and outputs responsibly, without creating a new privacy risk in the logs themselves.
• Enforce authorisation outside the model — never let the model decide who is allowed to do what.
Recovery Controls
• Provide kill switches, fallback modes, and tested rollback plans for every production model.
• Reassess controls after any change to the model, prompt, dataset, tool set, or infrastructure — a control validated once does not stay valid forever.
AI Security Testing, Red Teaming, and Evaluation
A model that scores well on ordinary accuracy or helpfulness benchmarks can still be highly insecure, because those benchmarks were never designed to simulate an adversary. Dedicated testing disciplines include:
• Threat-led and adversarial testing — simulating a specific attacker's goals rather than generic edge cases.
• AI red teaming — structured, often cross-functional attempts to break a system's safeguards before an attacker does.
• Penetration testing — the traditional infrastructure and application-layer testing that still applies to every AI deployment.
• Robustness and privacy evaluation — testing resilience to distribution shift and to membership-inference style probing.
• Supply-chain, agent tool-and-permission, RAG isolation, abuse-case, regression, and continuous evaluation — each targeting a different lifecycle stage described earlier.
Useful metrics include attack success rate, information leakage rate, policy-bypass rate, retrieval-authorisation failure rate, false-positive and false-negative rates, detection and containment time, unsafe tool-action rate, resource consumption, cost per adversarial request, and performance under distribution shift. No single benchmark, and no single red-team engagement, proves permanent security — a system re-tested only once and never again is a system whose next update ships untested.
Monitoring, Incident Response, and Recovery
Effective monitoring for an AI system collects more than server logs. Useful telemetry includes privacy-aware prompt and output logging, authentication and authorisation events, tool calls, retrieval events, model and prompt-version changes, unusual token or spending patterns, repeated policy-bypass attempts, model-file integrity checks, behavioural drift, and canary or tripwire mechanisms designed to catch manipulation early.
When an incident is confirmed, response steps generally include alert triage, model or prompt rollback, credential rotation, disabling affected tools, isolating the deployment, revoking model or agent access, restoring known-good artefacts, investigating which data was actually affected, notifying relevant stakeholders, preserving evidence, and running a post-incident review that feeds back into the control set.
AI incident response differs from general cybersecurity incident response mainly in scope — it has to account for data lineage, model versioning, and agent memory — but it should integrate with, not replace, an organisation's existing incident-response programme. Duplicating a separate “AI-only” response process tends to create gaps rather than closing them.
AI Model Security Frameworks and Standards
A handful of frameworks now anchor most AI security programmes. None of them is a certification, and following one voluntarily does not by itself create legal compliance or guarantee security — they are starting points, not finish lines.
NIST AI Risk Management Framework and Profiles
The NIST AI RMF 1.0 (January 2023) organises AI risk management into four functions — Govern, Map, Measure, and Manage — and works for any organisation regardless of sector. NIST later published the Generative AI Profile, NIST AI 600-1 (July 2024), which adds twelve risk categories specific to generative AI, and NIST AI 100-2e2025 (March 2025), a taxonomy of adversarial machine learning covering both predictive and generative AI attack types. Best used for: enterprise governance and a shared internal vocabulary. Primary audience: risk, governance, and security leadership. Limitation: voluntary and non-prescriptive — it tells you what to consider, not exactly how to implement it.
OWASP GenAI Security Project
The OWASP Top 10 for LLM Applications 2025 ranks the most critical risks specific to LLM-powered applications, from prompt injection to unbounded consumption. The OWASP Top 10 for Agentic Applications 2026 published in December 2025 extends this to autonomous agents, covering planning manipulation, tool misuse, identity abuse, and cascading multi-agent failures. Best used for: developers and application security teams building or reviewing a specific LLM or agent product. Limitation: focused on application-layer risk, not organisational governance.
MITRE ATLAS
MITRE ATLAS is a living, ATT&CK-style knowledge base of adversary tactics and techniques against AI-enabled systems, built from real-world observations and red-team demonstrations. Best used for: threat modelling and red-team scoping using a shared, structured vocabulary of attacker behaviour. Limitation: describes attacker techniques rather than prescribing specific defensive controls.
CISA, NSA, and International Guidance
The Joint Guidance on Deploying AI Systems Securely (April 2024), published by NSA, CISA, the FBI, and cybersecurity authorities from Australia, Canada, New Zealand, and the UK, focuses on best practices for operating externally developed AI systems inside existing IT environments. Best used for: organisations deploying, rather than building, AI systems. Limitation: deployment-focused, not a full lifecycle framework.
Google Secure AI Framework
Google's Secure AI Framework (SAIF), introduced in 2023, organises controls across four components — data, infrastructure, model, and application — and has since added guidance for agentic AI risks. Best used for: teams that want a conceptual risk map spanning the full AI stack. Limitation: vendor-authored, so some guidance assumes a particular cloud ecosystem.
Existing Security Standards
Established frameworks such as NIST's Cybersecurity Framework and ISO/IEC 27001 for information security management still apply underneath any AI-specific guidance — an AI system inherits every risk of the infrastructure it runs on.
A Compact Framework-Selection Guide
• Need enterprise-wide governance language → NIST AI RMF plus the Generative AI Profile.
• Building or reviewing a specific LLM or agent product → OWASP GenAI Top 10 lists.
• Running or scoping a red-team exercise → MITRE ATLAS.
• Deploying a third-party AI system inside existing infrastructure → the CISA/NSA joint guidance.
• Want a full-stack conceptual risk map → Google SAIF.
How to Build an AI Model Security Programme
First 30 Days
Build an inventory of every model, dataset, and connected tool in use, including tools employees have adopted informally. Assign clear ownership for each model. Classify each one by data sensitivity and autonomy.
Days 31–90
Write down security requirements for any new AI project before development starts. Add AI-specific questions to vendor and procurement reviews. Stand up basic monitoring for prompts, tool calls, and spending.
Months 4–12
Introduce structured red-team testing before major releases. Integrate AI incidents into the existing incident-response plan rather than building a separate one. Publish internal AI-use guidelines and run staff training on shadow-AI risk.
Ongoing Improvement
Reassess controls after every material model, prompt, or tool change. Track metrics over time rather than treating any single test as final. Review vendor and third-party model posture on a set schedule.
Smaller organisations do not need a dedicated AI security department to do this well — a single owner with clear authority, a short checklist, and a habit of re-testing after changes covers most of the risk. Useful roles as a programme matures include a model owner, data owner, product owner, security engineer, ML engineer, privacy or legal representative, risk owner, and an incident-response lead; in a small team, one person can hold several of these at once.
Common AI Model Security Mistakes and Myths
• “The model provider handles all security.” Providers secure their own infrastructure, not how you configure access, prompts, or tools around the model you deploy.
• “A private model cannot leak information.” Private deployment does not prevent memorised training data from surfacing in outputs, or prevent misuse by authorised insiders.
• “A strong system prompt prevents prompt injection.” Prompt wording alone cannot reliably separate trusted instructions from untrusted content, because both arrive as ordinary language.
• “Open-source models are always less secure.” Openness allows independent scrutiny; the real determinant is how carefully a given deployment is configured, not the licence.
• “Closed models are automatically secure.” Closed weights hide implementation detail but do nothing to prevent prompt injection, data leakage, or agent misuse at the application layer.
• “Red teaming once is enough.” A control validated before one release can fail silently after the next model, prompt, or tool update.
• “AI safety and AI security are the same.” Safety concerns intended behaviour without an attacker; security concerns deliberate manipulation. Both matter, and they need different tests.
• “Traditional cybersecurity controls are obsolete.” Identity management, network segmentation, and secure infrastructure remain necessary; they are simply no longer sufficient on their own.
• “The model should decide whether a user is authorised.” Authorisation has to be enforced outside the model, in code the model cannot talk its way around.
• “Removing obvious personal data solves every privacy risk.” Models can still infer or reconstruct sensitive attributes from patterns in the remaining data.
• “A model with no known CVE has no security risk.” Most AI-specific weaknesses are architectural or configuration issues, not the kind of flaw that receives a CVE.
• “The biggest model is the safest model.” Model scale has no consistent relationship with resistance to prompt injection, poisoning, or extraction.
The Future of AI Model Security
Several trends are already visible in current practice rather than pure speculation: greater agent autonomy and multi-agent workflows are pushing identity and permission design to the centre of AI security work, since a poorly scoped agent credential is now a bigger risk than a poorly worded prompt. Persistent memory across sessions is extending the attack surface for context poisoning. Model and data provenance — signing artefacts and tracking an AI bill of materials — is moving from a research idea toward standard supply-chain practice, mirroring what software supply-chain security went through a decade earlier.
Looking further out, informed forecasts rather than settled fact: continuous evaluation is likely to replace point-in-time red-team exercises as agentic systems ship more frequent updates; hardware-backed protections such as confidential computing may become more common for high-sensitivity inference; and dedicated AI-specific security operations functions may emerge inside larger security teams as the volume of AI-specific alerts grows. Regulation and procurement requirements, including frameworks like the EU AI Act, are pushing provenance and testing obligations onto vendors regardless of how mature their internal practice already is. The clearest overall direction is convergence: AI security is increasingly treated as one thread inside application, cloud, data, and identity security, rather than a separate discipline bolted on afterward.
FAQ
What is AI model security in simple terms?
AI model security means protecting an AI model and everything connected to it — training data, weights, APIs, and tools — from theft, tampering, and misuse. It covers the whole system around the model, not just the accuracy of its predictions.
Why is AI model security important?
Weak AI model security can expose sensitive data, let attackers steal costly-to-train models, manipulate high-stakes decisions, or trigger regulatory and reputational damage. The Samsung ChatGPT data leak in 2023 and the EchoLeak Copilot vulnerability in 2025 both show real, documented consequences.
What is the difference between AI security and AI safety?
AI security addresses deliberate manipulation by an attacker, such as poisoning or prompt injection. AI safety addresses whether a model behaves as intended even with no attacker present, such as avoiding harmful or biased outputs. The two overlap but require different tests.
What is an adversarial attack on an AI model?
An adversarial attack is a deliberate attempt to manipulate a model's training data, inputs, or weights to force an unintended outcome — for example, a small perturbation to an image that causes a classifier to misread it.
Can an AI model be hacked?
Yes. Models can be manipulated through poisoned training data, tricked through prompt injection or jailbreaking, or have their weights and behaviour copied through model extraction. The surrounding application and infrastructure can also be hacked using ordinary cybersecurity methods.
What is data poisoning?
Data poisoning is the deliberate manipulation of a model's training data so it learns a flawed, biased, or backdoored pattern. It typically requires the attacker to influence a data source the model or its pipeline trusts, such as a public dataset or user-submitted content.
What is model theft?
Model theft is the unauthorised copying of a proprietary model, either by directly accessing its weights file or by repeatedly querying its API and using the responses to train a substitute model, a process known as model extraction.
What is prompt injection?
Prompt injection is an attack where malicious instructions, either typed directly or hidden inside content the model reads, override the system's intended behaviour. It differs from SQL injection because natural language instructions cannot be sanitised the way structured queries can.
Are open-source models less secure?
Not inherently. Openness allows independent security review, which can improve security over time. The real determinant of security is how carefully a specific deployment is configured, monitored, and permissioned, not whether the model's weights are open or closed.
How do you secure an LLM?
Securing an LLM means combining input handling that treats untrusted content as data rather than instructions, output validation before actions are taken, least-privilege access for any connected tools, rate and cost limits, and ongoing adversarial testing rather than a one-time review.
How do you secure a RAG system?
RAG security requires document-level access control enforced before retrieval, treating all retrieved content as untrusted rather than authoritative, and monitoring for retrieval-authorisation failures such as one tenant's assistant surfacing another tenant's documents.
What security risks do AI agents create?
Agents add goal hijacking, tool misuse, and excessive agency to standard model risks, because an agent can take real, sometimes irreversible actions rather than just generate text. Multi-agent systems add the risk of one compromised agent cascading failure to others.
What is AI red teaming?
AI red teaming is structured, adversarial testing designed to find ways an AI system's safeguards can be broken before a real attacker finds them, covering prompt injection, jailbreaking, data leakage, and unsafe tool actions.
Which AI security framework should an organisation use?
Most organisations benefit from combining several: the NIST AI RMF and Generative AI Profile for governance, the OWASP GenAI Top 10 lists for application-level risk, and MITRE ATLAS for structured threat modelling and red-team scoping.
Who is responsible for AI model security?
Responsibility is typically shared across a model owner, data owner, security engineer, and ML engineer, with risk or legal representation for regulated data. Smaller organisations can cover this with one accountable owner and a clear escalation path rather than a dedicated department.
How often should AI models be security-tested?
Models should be re-tested after any material change to the model, prompt, dataset, or connected tools, not just once before initial release. A control validated on one version can fail silently on the next without further testing.
Key Takeaways
• AI model security protects the model and its entire surrounding system — data, weights, APIs, tools, and identities — not just prediction accuracy.
• The attack surface spans the full lifecycle: data acquisition, training, evaluation, deployment, inference, monitoring, and decommissioning.
• Major risks include data poisoning, backdoors, prompt injection, jailbreaking, model theft, and sensitive-information disclosure.
• Foundation models, LLMs, RAG systems, and agents each layer new risks on top of classic ML security concerns.
• Defence in depth — preventive, detective, responsive, and recovery controls across every layer — replaces reliance on any single safeguard.
• Testing has to be adversarial and ongoing; benchmarks that measure quality do not measure security.
• Monitoring and incident response for AI systems should integrate with, not duplicate, existing cybersecurity operations.
• Frameworks such as the NIST AI RMF, the OWASP GenAI Top 10 lists, and MITRE ATLAS provide shared vocabulary, not a substitute for an actual programme.
• The highest-value first step for most organisations is a plain inventory of models, data, tools, and ownership.
Actionable Next Steps
1. Inventory every model, dataset, and connected tool currently in use, including tools adopted outside official channels.
2. Classify each model by data sensitivity, autonomy, and potential impact to prioritise where controls go first.
3. Threat-model each high-priority system, listing realistic attackers, their goals, and their likely access level.
4. Assign a named owner for every model and dataset so security questions have a clear point of contact.
5. Enforce least-privilege access for any agent or service account, and remove standing credentials it does not need.
6. Add document-level access control before RAG retrieval, and treat all retrieved content as untrusted input.
7. Set rate, token, and spending limits on every inference endpoint before it goes into production.
8. Run adversarial and red-team testing before major releases, measuring attack success rate and policy-bypass rate, not just quality scores.
9. Stand up monitoring for prompts, tool calls, and spending, with alerting tuned to catch policy-bypass attempts.
10. Integrate AI incidents into your existing incident-response plan rather than building a separate, disconnected process.
11. Re-test controls after every material change to a model, prompt, dataset, or tool, since validation does not carry forward automatically.
Glossary
Adversarial example: An input deliberately modified, often with small changes invisible to a human, to cause a model to produce an incorrect output.
Adversarial machine learning: The study of how machine learning systems can be attacked and how to defend them, covering poisoning, evasion, and privacy attacks.
AI agent: A system that uses a model to plan and take actions, often through connected tools, rather than only generating text.
AI model: A trained system, represented as a set of weights, that produces predictions, recommendations, or generated content from input data.
Attack surface: Every point where an attacker could interact with or influence a system, including data, APIs, tools, and infrastructure.
Backdoor: A hidden pattern trained into a model so it behaves normally on typical input but produces attacker-chosen output on a specific trigger.
Data poisoning: Deliberately manipulating training data so a model learns a flawed, biased, or backdoored pattern.
Embedding: A numerical representation of text, images, or other content that captures meaning in a form a model can process mathematically.
Evasion attack: An attack that manipulates an input at inference time, rather than training time, to cause a model to misclassify it.
Foundation model: A large model trained on broad data that can be adapted to many downstream tasks through fine-tuning or prompting.
Inference: The process of running a trained model on new input to produce an output, as opposed to training the model.
Jailbreak: A technique used to get a model to ignore its safety training and produce content or actions it was designed to refuse.
Large language model: A model trained on large volumes of text to understand and generate human-like language.
Membership inference: An attack that determines whether a specific record was part of a model's training data.
Model extraction: Rebuilding a functional copy of a proprietary model by repeatedly querying it and training a substitute on the responses.
Model inversion: Reconstructing approximate training examples from a model's outputs or internal parameters.
Model poisoning: Manipulating the training process or a model's weights directly, rather than its training data, to introduce a flaw.
Model provenance: A verifiable record of where a model's data, weights, and code came from and how they changed over time.
Model registry: A system used to catalogue, version, and track trained models before and after deployment.
Model weights: The learned numerical parameters that define what a trained model has learned and how it produces output.
Multimodal model: A model that processes more than one type of input, such as text, images, and audio, together.
Prompt injection: An attack where instructions embedded in user input or retrieved content override a system's intended behaviour.
RAG: Retrieval-augmented generation: a technique where a model's response is grounded in content retrieved from an external source at query time.
Red teaming: Structured, adversarial testing designed to find ways a system's safeguards can be broken before a real attacker does.
Robustness: How well a model maintains performance when faced with noisy, shifted, or adversarial input.
Supply-chain attack: An attack that compromises a system through a trusted third-party component, such as a pre-trained model or dependency.
System prompt: The instructions given to a model by the developer, intended to define its behaviour before any user input is added.
Threat model: A structured description of a system's assets, likely attackers, their goals, and the paths they could use to reach those goals.
Vector database: A database optimised to store and search embeddings by similarity, commonly used to power retrieval-augmented generation.
Sources & References
NIST. “Artificial Intelligence Risk Management Framework (AI RMF 1.0).” National Institute of Standards and Technology, January 26, 2023. https://www.nist.gov/itl/ai-risk-management-framework
NIST. “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1).” National Institute of Standards and Technology, July 26, 2024. https://www.nist.gov/itl/ai-risk-management-framework
Vassilev, A., Oprea, A., Fordyce, A., Anderson, H., Davies, X. and Hamin, M.. “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2e2025).” National Institute of Standards and Technology, March 2025. https://www.nist.gov/publications/adversarial-machine-learning-taxonomy-and-terminology-attacks-and-mitigations-0
OWASP GenAI Security Project. “OWASP Top 10 for LLM Applications 2025.” OWASP Foundation, 2025. https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/
OWASP GenAI Security Project. “OWASP Top 10 for Agentic Applications for 2026.” OWASP Foundation, December 9, 2025. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
OWASP Foundation. “Prompt Injection.” OWASP Foundation, 2025. https://owasp.org/www-community/attacks/PromptInjection
MITRE. “MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems).” The MITRE Corporation, 2025. https://atlas.mitre.org/
NSA Artificial Intelligence Security Center, CISA, FBI, ASD ACSC, CCCS, NCSC-NZ, NCSC-UK. “Deploying AI Systems Securely.” U.S. National Security Agency / CISA, April 15, 2024. https://www.cisa.gov/news-events/alerts/2024/04/15/joint-guidance-deploying-ai-systems-securely
Google. “Introducing Google's Secure AI Framework.” Google, June 2023. https://blog.google/innovation-and-ai/technology/safety-security/introducing-googles-secure-ai-framework/
Google Safety Center. “Google's Secure AI Framework (SAIF).” Google, 2026. https://safety.google/intl/en_us/safety/saif/
Mark Gurman. “Samsung Bans Generative AI Use by Staff After ChatGPT Data Leak.” Bloomberg, May 2, 2023. https://www.bloomberg.com/news/articles/2023-05-02/samsung-bans-chatgpt-and-other-generative-ai-use-by-staff-after-leak
Mack DeGeurin. “Oops: Samsung Employees Leaked Confidential Data to ChatGPT.” Gizmodo, April 6, 2023. https://gizmodo.com/chatgpt-ai-samsung-employees-leak-data-1850307376
The Hacker News. “Zero-Click AI Vulnerability Exposes Microsoft 365 Copilot Data Without User Interaction.” The Hacker News, June 2025. https://thehackernews.com/2025/06/zero-click-ai-vulnerability-exposes.html
Dark Reading. “Researchers Detail Zero-Click Copilot Exploit 'EchoLeak'.” Dark Reading, June 12, 2025. https://www.darkreading.com/application-security/researchers-detail-zero-click-copilot-exploit-echoleak
Sentra. “EchoLeak (CVE-2025-32711): What the Microsoft Copilot Prompt Injection Vulnerability Means for Your Data.” Sentra, June 2026. https://sentra.io/blog/copilot-echoleak-prompt-injection
Promptfoo. “MITRE ATLAS.” Promptfoo, 2026. https://www.promptfoo.dev/docs/red-team/mitre-atlas/
OECD.AI. “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2e2025).” OECD.AI Policy Observatory, 2026. https://oecd.ai/en/dashboards/policy-initiatives/adversarial-machine-learning-a-taxonomy-and-terminology-of-attacks-and-mitigations-nist-ai-100-2e2025