▸How-to Support +1
Hostile support tickets with hidden commands
A bot that summarises and routes hostile tickets safely instead of acting on instructions hidden inside them.
Track 17 · Advanced
17.1
Keep your AI app secure
Defend what you build against prompt injection and abuse
1Overview
The chapter for people who ship AI, not just use it: privacy keeps your own secrets out of a chatbot, security defends a system you build against an attacker who feeds it hostile input. Taught as vulnerable → hardened pairs across the real threat model — prompt injection (direct and indirect), untrusted output, over-powered tools, and supply chain — mapped to the OWASP Top 10 for LLM Apps (2025).
In this chapter you will learn how to build AI applications that resist prompt injection and other forms of abuse by separating commands from untrusted data, enforcing strict access controls, and requiring human approval for high‑impact actions. You will be able to implement safe bots, audit trails, adversarial testing, and content sanitisation measures that keep credentials hidden, limit tool privileges, and treat all model input and output as potentially malicious. The techniques also cover supply‑chain verification, guard‑model screening, and layered defenses so that any injection is contained, detected, and blocked before it can cause damage.
1.1After this chapter you can
→Set a trust boundary between your instructions and untrusted input
→Treat model output as untrusted — never run it as code, HTML or SQL unsanitized
→Give agent tools least privilege and put a human in the loop for risky actions
→Vet your model/plugin supply chain, add guardrails, and red-team your own app
1.2What is prompt injection?
Prompt injection is when an attacker feeds hostile input that manipulates the AI’s instructions, causing it to behave contrary to your intended logic or reveal protected data.
1.3How can I protect untrusted output?
You must validate and sanitize any content generated by the model before using it downstream, ensuring it cannot be exploited to trigger unsafe actions or leak information.
1.4Why does supply‑chain matter for LLM apps?
A compromised component in your AI stack—such as a poisoned model or library—can introduce hidden vulnerabilities that attackers exploit, so you need to verify the integrity of every piece you integrate.
1.5The moves — weak → strong ladder
- 1Separate your instructions from untrusted input
- 2Defend against INDIRECT injection from retrieved content
- 3Screen input and output with a guard model
- 4Harden the system prompt against jailbreaks
- 5Keep secrets & access control OUT of the system prompt
- 6Treat model output as untrusted before it hits another system
- 7Give tools least privilege
- 8Require human-in-the-loop for high-impact actions
- 9Vet your model & plugin/MCP supply chain
- 10Layer guardrails, monitoring & red-team your own app
2Techniques
Learn
Trust boundaries & injection
Four layers of defense: separate instructions from input, screen retrieved content, guard with a classifier, harden the prompt
Separate your instructions from untrusted input The model cannot tell a command from data when they are mixed into one blob. Keep your rules in the system prompt and deliver everyone else's text as clearly-labelled, encoded data.
✕Instead of
Concatenate the user message (or a retrieved document) straight into the system prompt as one big string, so instructions and untrusted content sit side by side.
✓ Try this 💬 AI chat
Keep YOUR instructions in the system prompt. Deliver third-party content only inside clearly-labelled, encoded data blocks (e.g. wrapped in tags or a delimited field) that the prompt explicitly frames as data to process, not commands to obey.
Why it works: When commands and data are concatenated, the model has no reliable way to tell which is which — and an attacker phrases their data as a command. A clear boundary is the foundation every other defense builds on (OWASP LLM01).
Defend against INDIRECT injection from retrieved content The dangerous payload is rarely typed by your user — it is planted in a page, email, or PDF your system fetches and reads. Treat everything a tool returns as untrusted.
✕Instead of
Let a RAG app or agent obey instructions it finds inside a fetched web page, email, or PDF — because that text arrived through your own retrieval step, you trust it.
✓ Try this 💬 AI chat
Declare tool output, retrieved documents, and search results as untrusted data to REPORT, not commands to follow. Screen what a tool returns before the agent acts on it; never let fetched content silently change the plan.
Why it works: Indirect injection is the attack that scales: the adversary plants the payload in content your system reads on its own, so they never need access to your prompt. Labelling retrieved content as data is the core mitigation (OWASP LLM01).
Screen input and output with a guard model A cheap, separate classifier in front of (and behind) your main model catches the obvious attacks before they reach it — and catches leaks before they reach the user.
✕Instead of
Pass raw user input straight to the model and stream the raw output straight back to the user, with nothing watching either side.
✓ Try this 💬 AI chat
Pre-screen input with a small classifier for known injection patterns and obviously hostile requests; monitor outputs for leaked system prompts or exfiltrated data before they are shown. Block or flag what the screen catches.
Why it works: A small dedicated screen is cheap to run and catches the casual, high-volume attacks — freeing your main model and your team to focus on the subtler ones. It is one layer, not the whole defense (OWASP LLM01).
Harden the system prompt against jailbreaks A bare "be helpful" prompt invites probing. Named boundaries plus a refusal script and rate-limits resist casual jailbreaks — but the prompt is never the last line of defense.
✕Instead of
Rely on a one-line "be helpful and answer the user" system prompt, with no stated boundaries and no limit on how many times someone can probe it.
✓ Try this 💬 AI chat
State explicit boundaries and a clear refusal script in the system prompt, and rate-limit or ban repeat offenders who keep probing. Treat this as friction that raises the bar — never as the thing that holds the secret.
Why it works: Explicit, named boundaries plus rate-limits make casual jailbreaks far harder, but a determined attacker can still talk a model around its prompt — so this layer buys friction, not a guarantee (OWASP LLM01).
Output & agency
Don't trust model output, don't over-empower tools
Keep secrets & access control OUT of the system prompt A prompt can leak, and it cannot be trusted to enforce a rule. Credentials belong in a secret store; authorization belongs in deterministic code outside the model.
✕Instead of
Put an API key, a database password, or a rule like "only admins may delete records" directly in the system prompt and trust the model to keep it.
✓ Try this 💬 AI chat
Remove all credentials from prompts — load them server-side from a secret store the model never sees. Enforce authorization (who may do what) in deterministic code OUTSIDE the model, so a leaked prompt reveals nothing and bypasses nothing.
Why it works: A system prompt can be leaked by a clever jailbreak, and a model can be talked past a "rule" written in prose. Secrets and access checks must live where neither leaking nor persuading the model can reach them (OWASP LLM07 / LLM02).
Treat model output as untrusted before it hits another system Model output is just a string an attacker may have influenced. Encode and validate it the same way you would any user input before it touches HTML, SQL, a shell, or another service.
✕Instead of
Render the model output directly as HTML, eval() the code it returns, or interpolate it straight into a SQL query or shell command.
✓ Try this 💬 AI chat
Context-aware encode every output: HTML-escape before rendering, use parameterized queries for SQL, apply a strict Content-Security-Policy, and schema-validate structured output before any downstream system consumes it.
Why it works: Unsanitized model output is an injection vector into your OTHER systems — it becomes XSS, SQL injection, SSRF, or remote code execution exactly as untrusted user input would. Encode it at every boundary (OWASP LLM05).
Give tools least privilege A successful injection can only do what your tools allow. Expose narrow, purpose-built tools with the minimum scope — not an open-ended shell with broad admin rights.
✕Instead of
Hand the agent an open-ended shell tool and broad admin credentials so it "can do whatever the task needs."
✓ Try this 💬 AI chat
Expose only granular, purpose-built tools, each with the minimum scope to do its one job (e.g. "look up an order by id", not "run SQL"). Scope the credentials per task so a tool can touch only what it must.
Why it works: You cannot guarantee an injection never lands, but you CAN bound the blast radius: with least-privilege tools, the worst a successful injection achieves is whatever those narrow tools permit — and nothing more (OWASP LLM06).
Require human-in-the-loop for high-impact actions A confirmation step turns a successful breach into a blocked prompt. For anything irreversible or high-impact, the model proposes and a human approves.
✕Instead of
Let the agent send money, delete records, or email customers fully autonomously, with no human in the loop.
✓ Try this 💬 AI chat
Require explicit human approval — with a preview of exactly what will happen — for irreversible or high-impact actions (payments, deletions, outbound email). Rate-limit the lower-impact actions you do automate.
Why it works: Even if an injection slips through every earlier layer, a mandatory human approval on the dangerous action stops it cold: the breach becomes a suspicious prompt a person declines, not a wire transfer (OWASP LLM06).
Supply chain, guardrails & red-teaming
Trust your ingredients, watch the system, attack it yourself
Vet your model & plugin/MCP supply chain An untrusted model, adapter, or plugin can be backdoored. Trust your ingredients: use known registries, verify provenance, and keep a signed inventory of what you ship.
✕Instead of
Wire in a random fine-tune, LoRA adapter, or MCP server you found from an unverified source because it looked convenient.
✓ Try this 💬 AI chat
Use trusted registries; verify provenance with hashes or signatures before you load anything; keep a signed SBOM (software bill of materials) of every model, adapter, and plugin; and patch dependencies on a schedule.
Why it works: A model or plugin from an unverified source can carry a hidden backdoor or poisoned behaviour that no prompt-level defense will catch. Provenance and a signed inventory are how you trust what you actually shipped (OWASP LLM03 / LLM04).
Layer guardrails, monitoring & red-team your own app No single layer holds. Assume injection eventually succeeds, build defense-in-depth, log every tool call, and attack your own workflow before an outsider does.
✕Instead of
Ship once, assume the system prompt holds, and add no monitoring or adversarial testing.
✓ Try this 💬 AI chat
Stack defenses: input/output classifiers + filters + human-in-the-loop on risky actions + logging of every tool call. Then red-team your OWN workflow with planted injections before launch, and keep monitoring in production.
Why it works: Every individual layer can be bypassed, so the goal is containment and detection, not a perfect wall: assume an injection will land, make sure it cannot do much, and make sure you SEE it when it does (Google Security; OWASP LLM01).
3Lessons 7
3.1 Summarise tickets safely
A bot that receives support tickets, extracts the content, and returns a safe summary without executing any hidden commands.
Create a ticket summariser that treats incoming text as untrusted data
- Create an endpoint that accepts raw ticket text via POST
- Wrap the received text in a JSON object with a field such as untrusted_input
- Pass only the wrapped object to the language model, prompting it to produce a plain‑language summary
- Discard any model output that resembles code or commands and return only the summary string
- You'll see Ticket summaries appear in your UI while no hidden instructions are executed, even if the original ticket contained malicious prompts
- Takeaway Separating user‑provided text from executable actions prevents prompt injection at the data ingestion point
- Check How does wrapping incoming ticket text in a JSON field called untrusted_input prevent hidden commands from being executed during summarisation?
3.2 Activate the AI firewall to block malicious prompts
The SentinelOne AI firewall is a real‑time filter that scrubs incoming prompts and model outputs for injection attacks.
You will have enabled automatic blocking of adversarial prompts in your AI app.
- Open the SentinelOne console and navigate to AI Application Security.
- Select Real‑time AI firewall from the menu.
- Turn on the toggle labeled Block adversarial prompts.
- Save the configuration and deploy the updated policy to your application environment.
- You'll see A status indicator shows the firewall is active, and any test prompt containing injection syntax is rejected with a blocked‑prompt message.
- Takeaway Real‑time filtering prevents hostile inputs from reaching the model, providing a first line of defense against prompt injection.
3.3 Create an audit log that flags unusual tool calls
A logging layer that records each call a bot makes to external tools, including arguments and user context, and raises an alert when a call deviates from normal patterns.
Trace every tool usage and receive early warnings of suspicious activity
- Define a structured log schema containing timestamp, user_id, tool_name, args and result
- Instrument each tool‑calling function to write an entry to the persistent store using the defined schema
- Configure the system to send an alert (for example via email or Slack) whenever the anomaly detector flags a call
- You'll see A chronological log file shows each tool invocation and you get a notification when an out‑of‑pattern call occurs
- Takeaway Logging together with pattern monitoring provides a reliable audit trail and early detection for injection attempts
- Check Which elements of the structured log schema allow the system to recognise and alert on tool‑calling patterns that deviate from normal behaviour?
3.4 Create an audit log that flags unusual tool calls
SentinelOne Agentic AI Security provides searchable logs of every agent action, including tool invocations.
You will generate a log that automatically highlights tool calls that deviate from normal patterns.
- In the SentinelOne console, go to Agentic AI Security.
- Enable Audit logging for all agents.
- Define a rule under Unusual activity detection that flags any tool call whose parameters exceed a preset length or contain keywords like “delete” or “exec”.
- Apply the rule and verify it appears in the Log alerts view.
- You'll see The audit log shows entries for each tool call, with highlighted rows for calls that match the unusual‑activity criteria.
- Takeaway Continuous logging combined with anomaly rules lets you spot potential abuse of powerful tools before damage occurs.
3.5 Catch prompt injections before releasing your AI app
A short adversarial testing suite that injects crafted prompts into your bot and verifies that the system safely contains them.
An automated checklist validates all injection cases before shipping the AI application
- Create a list of representative injection strings such as hidden commands, malicious markup and API‑key leaks
- Write a test harness that sends each string to the bot and captures its response
- Assert that every response contains only safe output like summaries or error messages and never executes prohibited actions
- Integrate the harness into your CI pipeline so the build fails if any assertion is violated
- You'll see The CI run reports “passed” only when every injection test is safely handled, otherwise it blocks deployment
- Takeaway Embedding adversarial testing into the release process turns security assumptions into verifiable guarantees
- Check How does adding an adversarial injection test suite to your CI pipeline guarantee that a build fails whenever the bot mishandles a crafted malicious prompt?
3.6 Run a pre‑deployment test suite for prompt injection
SentinelOne’s AI Application Security can automatically test an app for prompt injection, jailbreaks, and data poisoning before it goes live.
You will execute a built‑in test suite that validates your app is resistant to common injection attacks.
- From the AI Application Security dashboard, select Pre‑production testing.
- Choose the Prompt Injection Test Pack and add it to the test queue for your application.
- Start the test run and wait for the results summary.
- If any test fails, open the reported issue, adjust the app’s input sanitisation logic, and re‑run the test until all checks pass.
- You'll see A report lists each injection vector tested; a green “All tests passed” status confirms the app is hardened against those attacks.
- Takeaway Automated pre‑release testing catches injection vulnerabilities early, integrating security into your CI/CD pipeline.
3.7 Enable secret scanning in GitHub to stop leaks before they start
GitHub Advanced Security’s secret protection feature scans code repositories for exposed credentials and other secrets.
You will configure secret scanning so that any committed secret triggers an alert and is blocked from merging.
- Open your repository on GitHub and go to Settings → Code security and analysis.
- Toggle Secret scanning to Enabled.
- Select the option to Block pull requests that contain detected secrets.
- Commit a test file containing a dummy API key (e.g.,
API_KEY=12345) and push it; verify GitHub creates an alert and prevents merging.
- You'll see GitHub displays a security alert for the dummy secret and blocks the pull request until the secret is removed.
- Takeaway Proactive secret scanning prevents credential exposure in source code, reducing attack surface before deployment.
4You’ll know it worked 26 checkable outcomes in this chapter
- ✓Agent processes a KB article containing an embedded instruction but does not act on that instruction, treating the article solely as reference data
- ✓Every tool invocation appears in the log with timestamp and arguments, and an alert fires when a high-impact or rarely-used tool is called
- ✓Launch is blocked until all 10 injection tests pass, and each test logs a failure if hijack detected
- ✓Model hash matches registry entry and is recorded in SBOM
- ✓The CMS receives only JSON that matches the schema; when validation fails, an error is logged and the payload is discarded
- ✓Payment action pauses and shows payee, amount, reason for approval before proceeding
- ✓Agent can view and edit only one record at a time; bulk export/delete requires approval
- ✓The résumé with malicious instructions is processed in isolation and does not modify any other candidate's records
26 outcomes in all — one per recipe below.
5FAQ, Tips & How-to 40
one problem, one solution, one action
Customer & client portals2
▸How-to Support +1
Bot can’t change anything on its own
Even if the bot is talked into something, its tools simply cannot issue refunds or change accounts without a human.
Knowledge & docs2
▸How-to Support +1
Hidden instructions in KB articles get ignored
Indirect injections hidden in your own knowledge base are caught before the agent follows them.
▸How-to HR / People +1
Hidden instructions in employee messages
The bot answers from policy safely, even when an employee's message hides an instruction.
Internal tools & ops15
▸How-to Operations +1
Ops agent runs destructive actions on its own
The agent can investigate freely, but every destructive action needs a human to confirm exactly what will happen.
▸How-to Operations +1
Unsure what actions my AI takes
A replayable audit trail and an early warning when the agent does something out of pattern.
▸How-to Operations +1
Want to catch prompt injections before launch
A short, repeatable adversarial pass that finds the gaps before a real attacker does.
▸How-to Founder +1
API keys showing up in chatbot prompts
A prompt-leak jailbreak exposes your instructions but none of your credentials.
▸How-to Founder +1
Potential input hijacks before release
A go/no-go security gate that catches an obvious hijack before customers (or attackers) do.
▸How-to Small biz +1
Admin rules can be bypassed by prompts
Permissions hold no matter what the model is persuaded to say, because the model never makes the call.
▸How-to Investor +1
Need to keep deal docs out of public AI tools
Confidential deal information is processed only inside a controlled environment, and you have a record of which tool you used and why it met the bar.
▸How-to Finance +2
Payments could run automatically
No payment ever leaves automatically; a person confirms exactly what is about to happen.
▸How-to Finance +2
AI can suggest overspending
Limits and approvals hold even when the model is persuaded otherwise, because it never enforces them itself.
▸How-to HR / People +1
Employees can’t pull salary info
Sensitive records stay gated by real authorization that the model cannot be argued past.
▸How-to HR / People +1
When a résumé tries to mess with other candidates
A poisoned résumé can only affect its own evaluation, never reach across to other candidates' data.
▸How-to HR / People
Recruiting while protecting candidate privacy
Candidate data is processed under a controlled arrangement, and your recruiting workflow handles that personal data responsibly from intake to archive.
▸How-to HR / People
AI just flags candidates
A recruiter reviews every shortlist decision; AI speeds up screening without replacing human judgment at the gate. Every fact in a candidate brief is source-confirmed.
▸How-to Physician
Never paste patient identifiers into public AI chatbots
A bright-line guardrail staff can follow without judgment calls in the moment, closing the most common accidental-leak path — pasting a real patient case into a general AI tool.
▸How-to Physician
Need a GDPR‑compliant rollout plan for a clinical AI tool
A clinical assistant that can be deployed under GDPR/data-protection obligations with a defensible paper trail, rather than one bolted on ad hoc after an incident.
Research & data tools4
▸How-to Scientist +1
Adversarial PDF tries to give new instructions
A research RAG that reads adversarial papers as content to summarise, never as commands to follow.
▸How-to Scientist +1
I can’t tell if a model is safe
You only run models whose origin you verified, with a record of exactly what you shipped.
▸How-to Scientist +1
Hidden instructions in search results
The agent reports what it found rather than blindly obeying instructions hidden in a search result.
▸How-to Investor +1
Need each memo number traced back to its source
Every metric in your investment memos and IC materials has a traceable, human-confirmed source — and the AI is used to draft and structure, never to originate numbers.
Content & marketing3
▸How-to Creator +1
AI‑generated text contains <script> tags
AI content renders as text, not as executable markup — closing a stored-XSS hole.
▸How-to Creator +1
Injected markup can run scripts in my AI chat widget
A second layer behind output-escaping: the browser itself blocks injected script execution.
▸How-to Creator +1
AI output that doesn’t match the expected shape is blocked
Only well-formed, expected output ever reaches the system that consumes it.
Forms, surveys & feedback1
▸How-to Finance +1
Invoice has hidden approve and pay command
The extractor reads hostile invoices safely instead of acting on text planted inside them.
CRM & sales3
▸How-to Sales +1
Incoming emails may hide malicious commands
Inbound emails are summarised and drafted from safely, never obeyed as commands.
▸How-to Sales +1
Need agents to touch only one CRM record each
A compromised sales agent can touch one record at a time, never dump or wipe the database.
▸How-to Sales +1
Unsure if AI‑generated emails are safe
Nothing leaves your domain without a person seeing exactly what will be sent and to whom.
▸How-to Everyone
Payments or deletions run automatically
A confirmation step turns a successful breach into a blocked prompt. For anything irreversible or high-impact, the model proposes and a human approves. Even if an injection slips through every earlier layer, a mandatory human approval on the dangerous action stops it cold: the breach becomes a suspicious prompt a person declines, not a wire transfer (OWASP LLM06).
▸How-to Everyone
Giving an AI a generic admin shell
A successful injection can only do what your tools allow. Expose narrow, purpose-built tools with the minimum scope — not an open-ended shell with broad admin rights. You cannot guarantee an injection never lands, but you CAN bound the blast radius: with least-privilege tools, the worst a successful injection achieves is whatever those narrow tools permit — and nothing more (OWASP LLM06).
▸How-to Everyone
Model output could be malicious
Model output is just a string an attacker may have influenced. Encode and validate it the same way you would any user input before it touches HTML, SQL, a shell, or another service. Unsanitized model output is an injection vector into your OTHER systems — it becomes XSS, SQL injection, SSRF, or remote code execution exactly as untrusted user input would. Encode it at every boundary (OWASP LLM05).
▸How-to Everyone
Putting API keys or admin rules in prompts
A prompt can leak, and it cannot be trusted to enforce a rule. Credentials belong in a secret store; authorization belongs in deterministic code outside the model. A system prompt can be leaked by a clever jailbreak, and a model can be talked past a "rule" written in prose. Secrets and access checks must live where neither leaking nor persuading the model can reach them (OWASP LLM07 / LLM02).
▸How-to Everyone
User prompts go straight to the model
A cheap, separate classifier in front of (and behind) your main model catches the obvious attacks before they reach it — and catches leaks before they reach the user. A small dedicated screen is cheap to run and catches the casual, high-volume attacks — freeing your main model and your team to focus on the subtler ones. It is one layer, not the whole defense (OWASP LLM01).
▸How-to Everyone
A plain ‘be helpful’ prompt gets probed
A bare "be helpful" prompt invites probing. Named boundaries plus a refusal script and rate-limits resist casual jailbreaks — but the prompt is never the last line of defense. Explicit, named boundaries plus rate-limits make casual jailbreaks far harder, but a determined attacker can still talk a model around its prompt — so this layer buys friction, not a guarantee (OWASP LLM01).
▸How-to Everyone
Content fetched from webpages or PDFs may hide attacks
The dangerous payload is rarely typed by your user — it is planted in a page, email, or PDF your system fetches and reads. Treat everything a tool returns as untrusted. Indirect injection is the attack that scales: the adversary plants the payload in content your system reads on its own, so they never need access to your prompt. Labelling retrieved content as data is the core mitigation (OWASP LLM01).
▸How-to Everyone
Instructions and user input run together
The model cannot tell a command from data when they are mixed into one blob. Keep your rules in the system prompt and deliver everyone else's text as clearly-labelled, encoded data. When commands and data are concatenated, the model has no reliable way to tell which is which — and an attacker phrases their data as a command. A clear boundary is the foundation every other defense builds on (OWASP LLM01).
▸How-to Everyone
Prompt injection gets through one guardrail
No single layer holds. Assume injection eventually succeeds, build defense-in-depth, log every tool call, and attack your own workflow before an outsider does. Every individual layer can be bypassed, so the goal is containment and detection, not a perfect wall: assume an injection will land, make sure it cannot do much, and make sure you SEE it when it does (Google Security; OWASP LLM01).
▸How-to Everyone
Using an unverified model or adapter
An untrusted model, adapter, or plugin can be backdoored. Trust your ingredients: use known registries, verify provenance, and keep a signed inventory of what you ship. A model or plugin from an unverified source can carry a hidden backdoor or poisoned behaviour that no prompt-level defense will catch. Provenance and a signed inventory are how you trust what you actually shipped (OWASP LLM03 / LLM04).
The same set on /recipes, filtered by tool and role.
6Videos 3
OWASP's Top 10 Ways to Attack LLMs: AI Vulnerabilities Exposed
A clear tour of the OWASP Top 10 risk map this chapter follows — the threats you are defending against, explained for builders.
What Is a Prompt Injection Attack?
The headline risk explained from first principles — direct vs indirect injection and why mixing instructions with untrusted data is the root cause.
Securing AI Agents: How to Prevent Hidden Prompt Injection Attacks
Defensive patterns for agents specifically — guarding against payloads hidden in retrieved content, with least privilege and screening.
7FAQ 8
What's the difference between this and the Privacy chapter?
Different threat, different side of the table. The Privacy chapter is about you as a USER keeping your own secrets out of a chatbot you type into. This chapter is about you as a BUILDER defending an AI system you ship against an attacker who feeds it hostile input — prompt injection and the rest. Privacy protects your data going in; security protects your application against text designed to subvert it.
What is prompt injection, and what is "indirect" injection?
Prompt injection is when untrusted text is interpreted as instructions and hijacks what your model does. DIRECT injection is the user typing the hostile instruction themselves. INDIRECT injection is more dangerous: the attacker plants the instruction inside content your system later reads on its own — a web page, an email, a PDF, a document in your RAG index — so it executes without the attacker ever touching your prompt. The core defense for both is to keep your instructions separate and treat all third-party content as data, not commands.
How do I stop a RAG app or agent obeying instructions hidden in a document?
Declare retrieved documents, tool output, and search results as untrusted DATA to report on, never as commands to follow — keep your real instructions in the system prompt, clearly separated. Screen what a tool or retrieval step returns before the agent acts on it, and use guardrails (a classifier in front of the model, output monitoring behind it) plus least-privilege tools so that even a payload that slips through cannot do much. Assume some will get through and contain the impact.
Can I just put "don't reveal your instructions" in the system prompt to keep secrets safe?
No. A system prompt can be leaked by a determined jailbreak, and a rule written in prose can be talked past — so the prompt is the wrong place for anything that must hold. Keep credentials out of the prompt entirely (load them server-side from a secret store) and enforce authorization in deterministic code outside the model. Then a leaked prompt reveals no secret and bypasses no check.
Why is treating the model's output as trusted dangerous?
Because model output is just a string an attacker may have shaped — and if you render it as HTML, eval() it, or interpolate it into SQL or a shell command, that string becomes XSS, SQL injection, SSRF, or remote code execution in your OTHER systems. The fix is to treat output exactly like untrusted user input: context-aware encode it (HTML-escape, parameterized queries), apply a strict CSP, and schema-validate structured output before anything downstream consumes it.
How much power should I give an AI agent's tools?
As little as the job needs. Expose granular, purpose-built tools each scoped to one task, rather than an open-ended shell with broad admin credentials — and require human approval (with a preview) for irreversible or high-impact actions like payments and deletions. Least privilege bounds the blast radius: even a successful injection can only do what your narrow tools permit, and the dangerous actions still need a human to say yes.
Is a single defense enough, or do I need layers?
You need layers. No single mitigation is reliable on its own — a determined attacker eventually finds the gap — so the goal is defense-in-depth aimed at containment and detection: input/output classifiers, content filters, least-privilege tools, human-in-the-loop on risky actions, and logging of every tool call, combined with red-teaming your own workflow. Assume an injection will eventually land, make sure it cannot do much, and make sure you see it when it does.
Can I download any open model or plugin and trust it?
No. An untrusted model, fine-tune, adapter, or plugin/MCP server can be backdoored or carry poisoned behaviour that no prompt-level defense will catch. Use trusted registries, verify provenance with hashes or signatures before loading anything, keep a signed SBOM of every model and plugin you ship, and patch dependencies. Trust your ingredients the same way you trust the rest of your supply chain.
8Glossary 11 terms
Show the 11 terms
The attacks
Indirect prompt injection- An injection where the hostile instruction is hidden inside content your system fetches and reads (a page, email, PDF, or RAG document) rather than typed by the user — so it runs without the attacker touching your prompt.
Jailbreak- A crafted message that talks a model around its stated boundaries to make it do something it was instructed not to. Named boundaries and rate-limits raise the bar but never guarantee a stop.
System-prompt leakage- When an attacker extracts your hidden system prompt — exposing any secrets, credentials, or access rules you (wrongly) stored in it. The reason secrets and auth must live outside the prompt.
Data / model poisoning- Tainting the data a model learns from — training data, fine-tunes, or a RAG index — so it behaves the way the attacker wants. Defended by vetting sources and verifying provenance.
The weaknesses
Improper output handling- Treating model output as trusted and passing it unsanitized into another system, where it becomes XSS, SQL injection, SSRF, or RCE. Fixed by encoding and validating every output.
Excessive agency- Giving an agent more capability, permission, or autonomy than it needs — an open-ended shell, broad credentials, no approval gate — so a successful injection can do real damage. Fixed by least privilege.
The defenses
Trust boundary- The line your design draws between content it controls (your instructions) and untrusted input (user text, tool output). Keeping the two clearly separated is the foundation of injection defense.
Guardrail- A check around the model — an input classifier, an output monitor, a content filter — that catches hostile input or leaked data before it does harm. One layer of defense-in-depth.
Least privilege- Granting each tool and credential the minimum scope to do its one job, so the blast radius of any breach is bounded to exactly what that narrow tool permits.
Human-in-the-loop (HITL)- Requiring explicit human approval (with a preview) before an irreversible or high-impact action runs — turning a successful breach into a suspicious prompt a person declines.
Red-teaming- Adversarially testing your OWN system before launch — planting injections in the inputs and content it reads — to find and fix the gaps before an outsider does.
9See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.