Prompt Injection Defense Guide: Protect LLM Applications, RAG Systems, and AI Workflows

Overview

A prompt injection attack occurs when text supplied to an LLM attempts to change the instructions the application intended the model to follow. The text may come from a user, a document, a web page, an email, a support ticket, a database record, or the output of another model. It can look like an ordinary instruction, hidden markup, a request to reveal confidential context, or content designed to influence a downstream tool call.

There are two useful starting categories:

• Direct injection: a user places adversarial instructions in the message sent to the application. For example, a user may ask an assistant to ignore its task and reveal system instructions.
• Indirect injection: untrusted content retrieved or processed by the application contains instructions aimed at the model. A document in a RAG system, a web page, or a calendar description could tell an agent to send data, call a tool, or disregard its original task.

The central security assumption is simple: retrieved content and user-provided text are data, even when that data contains imperative language. They should not automatically become application instructions. Prompt engineering can make that distinction clearer, but it cannot replace authorization, isolation, validation, logging, and review.

Before reviewing a prompt, map the complete workflow: input sources, retrieval steps, model calls, memory, tools, external systems, outputs, and people who approve or consume results. A narrow chatbot that only drafts text has a different risk profile from an agent that can modify records, send messages, execute code, or access private files.

For implementation context, pair this review with a RAG workflow review and a tool-use and function-calling review. Those components determine where untrusted content enters and what the model can cause to happen.

Checklist by scenario

1. User-facing chat and direct prompting

Start with the public input boundary. Assume that a user can ask the model to change roles, expose hidden instructions, reproduce private context, or perform an action outside the stated task.

• Define the assistant's allowed purpose and prohibited actions in application logic, not only in a system prompt.
• Separate user content from developer and system instructions using clear message roles and explicit delimiters.
• Tell the model to treat requests for hidden prompts, credentials, internal notes, or unrelated private data as untrusted or unauthorized.
• Limit the information placed in the context window. Do not include secrets merely because the model might need them later.
• Test role-confusion attempts, instruction overrides, encoded requests, long distracting inputs, and requests that combine a legitimate task with an unsafe one.
• Apply output checks before displaying sensitive content or passing it to another component.

A system prompt can establish instruction hierarchy, but it should not be treated as a security boundary. Authentication, authorization, data filtering, and server-side policy checks should make the final decision.

2. RAG applications and uploaded documents

RAG systems create an indirect-injection surface because retrieved text is inserted into a model context. A malicious instruction can be stored in a document, hidden in a page, or added to a knowledge source by a user who should not control application behavior.

• Classify every data source by trust level, owner, sensitivity, and expected format.
• Preserve source identity and permissions through ingestion, chunking, retrieval, and generation.
• Tell the model that retrieved passages are reference material and cannot change application rules or authorize actions.
• Use retrieval filters so a user cannot obtain documents merely because the model was persuaded to request them.
• Scan documents for suspicious instruction patterns, but do not rely on scanning alone; benign text can contain commands and attacks can avoid simple keywords.
• Require citations, source identifiers, or evidence fields when the application needs to distinguish supported answers from unsupported claims.
• Keep high-risk actions outside the retrieval-and-answer loop until a separate authorization step succeeds.

When a document says “ignore previous instructions,” the safe behavior is not necessarily to reject the entire document. The application can treat that sentence as content, continue extracting relevant facts, and prevent it from changing permissions or workflow rules.

3. Agents with tools and external actions

The consequences of injection increase when a model can call APIs, send email, create tickets, update records, browse websites, run code, or access files. Review each tool as if the model might call it for the wrong reason.

• Give each tool the narrowest permissions and input scope it needs.
• Validate tool arguments with schemas and server-side business rules.
• Use read-only tools where possible, and separate data lookup from state-changing operations.
• Require confirmation for irreversible, external, financial, privacy-sensitive, or high-impact actions.
• Prevent the model from selecting arbitrary URLs, file paths, commands, recipients, or query fragments without validation.
• Log the user request, relevant context identifiers, proposed tool call, authorization result, and final outcome.
• Set time, rate, quantity, and destination limits to contain repeated or unexpected calls.

Structured output helps downstream code parse a response, but valid JSON is not proof that the request is safe. See the guide to reliable structured outputs, then add authorization and validation around every consequential action.

4. Automated workflows and model-to-model pipelines

In an automated pipeline, one model's output may become another model's instruction. That creates an indirect trust transition that is easy to overlook.

• Label outputs as untrusted until they pass validation.
• Use fixed schemas and permitted-value lists between workflow stages.
• Pass only the fields needed by the next step instead of forwarding full transcripts.
• Prevent generated text from directly becoming executable code, SQL, an email recipient, or an access-control decision.
• Add a human review gate for exceptions, low-confidence results, or actions outside normal thresholds.

A human-in-the-loop workflow is most useful when the approval condition is specific: what is being approved, which evidence supports it, and what happens if the reviewer rejects it.

Attack-pattern reference table

What to double-check

Use the following risk-assessment checklist before release and whenever the workflow changes:

1. Inputs: Have you listed every user, file, web, database, integration, and model-generated input?
2. Trust: Is each input labeled as trusted, conditionally trusted, or untrusted? Can the model tell the difference?
3. Secrets: Could a prompt, retrieved passage, error message, log, or output expose credentials or personal information?
4. Permissions: What can the application do if the model is manipulated? Are permissions enforced outside the model?
5. Tools: Are tool names, arguments, destinations, and quantities validated independently?
6. Outputs: Can generated text be rendered as HTML, executed as code, interpreted as SQL, or sent to an external party?
7. Recovery: What happens after a rejected tool call, timeout, malformed output, or suspected injection? Does a retry accidentally repeat the unsafe action?
8. Observability: Can the team reconstruct the relevant request, retrieved sources, model response, decision, and action without storing unnecessary sensitive content?
9. Evaluation: Are attack tests part of the same regression suite as quality tests?

For each test, record the input source, expected safe behavior, observed response, tool calls, data exposure, severity, and remediation owner. A practical test-case template is:

Scenario:
Input source:
Trust classification:
Adversarial content:
Expected model behavior:
Expected tool behavior:
Expected data exposure: none / limited / approved
Observed response:
Observed tool calls:
Pass or fail:
Follow-up control:

Run these cases against both normal and adversarial variants. Prompt changes should be evaluated for safety and usefulness, not only whether the model refuses a particular phrase. The prompt testing and evaluation checklist can help organize comparative tests.

Common mistakes

• Relying on a longer system prompt: More instructions do not create authorization boundaries or stop a compromised tool from acting.
• Trusting retrieved content because it came from an internal index: Internal sources can be stale, incorrectly permissioned, user-editable, or deliberately contaminated.
• Filtering only for obvious phrases: Keyword blocks can miss paraphrases and may reject harmless content. Use layered controls.
• Giving an agent broad credentials: The model should not inherit a service account's full power when a narrower operation is sufficient.
• Checking the final answer but not tool calls: A polished response can conceal that an unsafe action already occurred.
• Automatically retrying every failure: Retries can repeat a side effect. Classify errors and make state-changing actions idempotent or approval-based.
• Testing only clean prompts: Security tests should include documents, URLs, attachments, long context, encoded text, and mixed legitimate-malicious requests.
• Logging everything: Detailed logs are useful, but they can become another sensitive data store. Redact and restrict access.

When to revisit

Revisit this checklist before releasing a new AI workflow, connecting a new data source, adding a tool, changing retrieval or memory behavior, switching models, or changing the system prompt. Repeat focused red-team tests after a significant architecture change and include the results in the release review.

For ongoing maintenance, schedule a periodic review at a cadence appropriate to the application's risk. Refresh the attack cases, permissions, provider controls, framework settings, and output validators rather than assuming that an old test suite covers a new design. Seasonal planning cycles are also a useful reminder to review ownership, escalation paths, and human approval capacity.

A practical next step is to draw the workflow on one page, mark every untrusted input and side effect, then run three tests for each boundary: an instruction override, a data-extraction attempt, and a tool-manipulation attempt. Fix the highest-impact control gaps first. In most applications, that means reducing permissions, removing unnecessary sensitive context, validating tool arguments, and adding approval before irreversible actions. Use safe error-handling patterns so failures do not create a second path around those controls.