How to Prevent Prompt Injection in AI Agents

Your agent summarizes a vendor document. A line buried in that file tells it to refund every open ticket and send confirmations to an external address. The agent follows it and starts issuing refunds.

The failure happened at the whiteboard, before the document arrived.

Prompt injection holds the top rank in OWASP Top 10 for LLM Applications in both 2023 and 2025 because every app that ingests outside text is exposed and no single model fix closes it. Model robustness has improved. GPT-6 Astra reports 99.99% defender success on instruction hierarchy and 99.79% on indirect prompt injection on internal evaluations, up from 96.23% for the prior generation.

External Gray Swan testing across 1,810 attacks still shows an 8.5% chance of at least one success in 15 attempts for Astra. Gains matter and stay insufficient as sole defense when an agent holds state-changing tools.

Prevention sits in agent design. Seven architectural decisions constrain what a compromised model turn can reach.

A platform enforces the generic mechanics. Product teams retain the domain policy a platform cannot decide.

1. Decide where instructions end and data begins

The model sees instructions and data as one token stream. Concatenate them into a single string and attacker text inherits developer authority.

Keep system and developer instructions in their native role, pass user messages as user, and wrap every tool return or retrieved document in an explicit untrusted zone with provenance. OWASP calls this segregating external content. Microsoft spotlighting and AWS prescriptive guidance describe the same practice: label data as data and tell the model to reference it, not obey it.

Concrete steps:

• Tag every external byte with source, trust level, and retrieval time, and carry the tag through the pipeline
• Delimit untrusted blocks with markers the model is trained to treat as data
• For high-risk flows, isolate reading: a quarantined reader model with no tool access sees hostile content and returns a validated extract to the privileged planner

This dual LLM pattern appears in Simon Willison's writing and in Beurer-Kellner et al. among six patterns that offer verifiable resistance by guaranteeing a model that has ingested untrusted input cannot trigger consequential actions.

Platform work: enforce role separation, delimiters, provenance metadata, and the quarantined reader behind a constrained interface. Product work: classify what is trusted. Only the product can declare that a CRM note, ticket attachment, or third-party API response is untrusted by default.

2. Decide what the agent can touch

Severity tracks agency. An agent with five narrow read tools has a different blast radius from one that holds private-data access, browsing of untrusted content, and an egress tool in one context. OWASP and the Agent Security Cheat Sheet call that combination the lethal trifecta and recommend removing one leg by design.

Design choices:

• Scope tools per task. A summarization task gets no delete tool. A research task gets no write access
• Issue short-lived, task-scoped credentials for the tools that remain
• Keep sensitive data, untrusted exposure, and external communication out of the same trust context. If browsing is required, strip credentials from that context or remove its exfiltration channel

Two restricted patterns enforce this:

• Action selector: model chooses from a fixed set of predefined actions and never ingests untrusted feedback in the same decision
• Plan then execute: generate a fixed plan before any untrusted content is seen, then run it without letting tool outputs alter control flow

For MCP surfaces, OWASP MCP03:2025 Tool Poisoning warns a single poisoned schema that remaps a benign verb to a destructive one can compromise every tenant that trusts it. Mitigations include immutable registries, signed manifests, and version pinning.

Platform work: tool registration, credential scoping, registry enforcement, allowlists of approved MCP servers, manifest pinning by hash, signature verification. Product work: the authorization model. The platform cannot know that refunds above a threshold need a manager, that a customer in one region cannot receive an SMS, or which roles may expose capabilities to external clients via MCP.

3. Decide what a valid tool call looks like

A correctly scoped tool can still be abused with arbitrary arguments. Validate every invocation in code that attacker text never controls.

Require strict schemas for every tool input and output. Validate outside the model with Pydantic or JSON Schema. Encode semantic invariants as policy-as-code:

• Archive cannot map to DELETE
• Export cannot target an arbitrary domain
• Bulk action cannot exceed a defined volume

Run checks in CI and again at a runtime policy decision point before execution. Bind each invocation to a schema hash and provenance metadata.

OWASP frames this as defining expected output formats and enforcing complete mediation, where the model is treated as an untrusted client. For MCP, sign schemas, use content-addressable hashes, log the hash per call, and require human approval for schema changes that alter semantics.

For tool responses, prefer structured JSON and reject free text that does not match the expected shape. Schema validation forces remaining attacks into observable channels.

Platform work: schema validation, policy enforcement in the call path, server-side access control. Product work: schema definitions and policy thresholds that encode business rules.

4. Decide what requires a human

Some actions should never happen on model authority alone. OWASP lists requiring human approval for high-risk actions as a core mitigation.

Gate by impact:

• Reads and drafts can be automatic
• State-changing or external actions such as payments, deletions, permission changes, outbound messages, and external HTTP requests wait for explicit confirmation outside the LLM context

Approval should carry original user intent plus proposed tool and arguments, so a reviewer sees drift from the task. Keep the gate outside model interaction so an injected instruction cannot approve itself. Enforce limits on tool chain length, cost, retry, and capability scope alongside the approval, and run the action through server-side authorization.

Platform work: approval routing, token and chain limits, isolation of privileged tools into a context external MCP responses cannot reach. Product work: calibrate what counts as destructive in the domain and what evidence a reviewer needs.

5. Decide what leaves the system

Output filtering catches the common payoff: exfiltration, leaked secrets, credential reuse. OWASP lists input and output filtering and groundedness checks among mitigations.

Design choices:

• Validate outputs against policy before return or execution
• Inspect tool-call arguments for exfiltration patterns such as external URLs built from private data
• Sanitize rendering: strip or rewrite outbound URLs so the model cannot smuggle data inside a markdown image link, sanitize HTML and markdown before display, scan for credential-like strings and PII before egress
• Enforce default-deny egress allowlists for browsing and code execution sandboxes

A successful injection that tries to POST to an unapproved endpoint still fails at the network layer.

Platform work: input guardrails, output guardrails, tool-argument inspection, with a quarantined model inspecting without tool authority. Product work: data classification and the allowlist of legitimate external destinations.

6. Decide what you can see later

Logging is a mitigation and the evidence base for every other mitigation.

Log every prompt, every retrieved source with trust level, every tool call with inputs and outputs, and every guardrail decision and schema hash used, with timestamps and identity. Carry provenance through the trace.

Add alerting for anomalous sequences: prompt-injection markers, unexpected tool chains, sudden volume spikes on sensitive actions, or a shift in guardrail refusal distribution. Keep a kill switch that can revoke credentials or pause the agent without a redeploy.

Platform work: structured decision metadata, evaluations in CI, dashboards and alerts. Product work: retention policy and tenant isolation for logs.

7. Decide how you will break it

Risk changes with each model update, schema change, and new data source. Static review misses it. Testing needs to be continuous.

Red-team the agent on a schedule and in CI with a taxonomy that separates intent, technique, evasion, and input surface. OWASP Secure Agent Playbook describes a test matrix that prioritizes:

• Direct input with high-impact intents
• Indirect surfaces such as file uploads and RAG
• Multi-chain coercion across chained LLM calls
• Evasion sweeps with obfuscation, encoding, and payload splitting

Test attacker text typed by the user and attacker text hidden in tool outputs. Include documents, emails, web pages, and tool results with base64 and translated payloads and zero-width characters.

Measure attack success rate across releases. Maintain separate suites for multi-turn attacks and cross-agent propagation.

Platform work: harness, injection corpora, measurement per release. Product work: acceptance criteria for which intents count as failure.

Building with these decisions on Ginger Labs

These seven decisions define a platform and product split. Users describe an outcome, the agent does the work inside the customer's product experience.

Ginger Labs manages the agent layer and the managed MCP infrastructure for MCP as a service:

• Retrieval, evaluations, self-learning loops, and observability in the SDK
• Tool scoping, credential lifecycle, schema validation, and server-side access enforcement in the runtime
• Provenance tagging, delimiting, and dual-model isolation for untrusted content
• Input, output, and action guardrails with deterministic policy enforcement
• Structured logging, evaluation harnesses, and red-team suites that run before and after launch
• For MCP exposure: manifest and version pinning by hash, signature and hash verification before use, allowlists, and re-scan on version change

The customer retains what a platform cannot infer:

• Product API and data model
• Domain rules and workflow definitions
• User permissions and tenant boundaries
• Allowed actions and the customer-facing experience
• Business definition of a correct result

That includes which capabilities are exposed through MCP, how access is governed per tenant and role, and which data classes require filtering before egress.

A practical validation is one valuable workflow traced end to end in a 20-minute sandbox demo. Walk the seven decisions against that trace and note which controls the platform already enforces.

Sources

• OWASP LLM01:2025 Prompt Injection
• OWASP LLM Prompt Injection Prevention Cheat Sheet
• OWASP AI Agent Security Cheat Sheet
• OWASP MCP03:2025 Tool Poisoning
• OWASP Secure Agent Playbook: Prompt Injection Testing
• OpenAI GPT-6 Astra System Card: Prompt Injection, September 2026
• Anthropic Claude Fable 5.1 and Mythos 5.1 System Card, September 1, 2026
• Beurer-Kellner et al., Design Patterns for Securing LLM Agents against Prompt Injections, arXiv:2506.08837v3
• AWS Prescriptive Guidance: Security for Agentic AI on AWS
• Simon Willison: Prompt Injection taxonomy and Dual LLM pattern