Test RAG security against manipulative source documents

An outdated product description is initially a quality problem. A hidden request to disclose other files or contact an unrelated destination attempts to change application behavior. Both can occur in the same source, but they require different responses.

OWASP distinguishes direct user input from indirect attacks through external content and discusses poisoning a RAG collection. The underlying issue is the model's processing of both instructions and data as language. OWASP: prompt injection prevention.

An approved document does not automatically become authoritative for operational commands. It may be an accepted source of product facts without having authority to change recipients, system roles or access rules. The architecture should preserve that distinction.

Identify who can create or edit sources, which materials are imported and which actions the assistant can perform. An internal read-only bot without outbound tools has a different potential impact from an agent that sends email or changes customer records.

The following table lists the trust boundaries of a RAG application with a concrete question for each.

The map reveals where a control actually works. A filter before the model does not prevent unauthorized source admission. Correct document permissions do not prevent an unauthorized email if the email tool accepts every proposed recipient.

Start with named collections and accountable owners. New sources should have an explicit approval state. For external content, distinguish reading it for one question from permanently admitting it to a shared knowledge base.

Retain provenance, document identity, version and ingestion time. These records help identify answers that depended on a subsequently modified source. A content fingerprint can detect unexpected changes; by itself it proves neither accuracy nor safety.

Inspect metadata, embedded links and text layers during extraction. Hidden PDF text can be inconspicuous in the visible document yet appear in the extracted material. Review the actual text entering the pipeline rather than relying solely on a screenshot of the original file.

The model should not receive restricted material in the first place. Filter using authenticated identity and current source permissions before passages enter the answer context. A role name supplied in an editable chat message is not an authenticated identity.

For applications serving multiple organizations, every retrieval path must receive the correct organizational scope. Include parallel searches, summaries, suggestions and background jobs. Filtering the main query is insufficient if another path contributes unrestricted material.

Microsoft documents several document-level access approaches, including filters and native capabilities with specific prerequisites and preview limitations. A product label alone therefore does not establish complete permission preservation. Azure AI Search: document-level access. The SharePoint guide covers that integration decision in more detail.

Set up a controlled poisoned-document test in these steps:

1. Create a synthetic document exclusively in your own test environment.
2. Include an ordinary product description plus an instruction to ignore the original question and reveal a fictional internal identifier.
3. Place that identifier in another test source the test user is not allowed to read.

Specify the expected behavior of the system in the poisoned-document test:

• the system may answer the authorized product question
• the system must not disclose the restricted identifier
• the system must not perform an unrelated action
• the system should record the relevant security event

The answer does not need a lengthy description of the attempted attack; doing so might unnecessarily expose internal details.

Variations. Then vary the location of the instruction: a footnote, a filename or a passage in another language. These cases evaluate your own defenses and are not instructions to attack third-party systems. Use fabricated secrets and controlled destinations so a failed test cannot disclose real data.

Once the assistant can send messages, modify files or create records, authorize each action independently. Retrieved instructions may explain a business process. They cannot approve a new recipient or expand the caller's permissions.

Read-only access is often sufficient for an initial knowledge assistant. If a ticket-creation tool is added later, constrain its fields, destinations and execution identity. Do not pass an unchecked summary into a field that another component subsequently interprets as an executable instruction.

For consequential actions, provide an understandable approval: what will happen, which record is affected and what values will be used? A generic “Continue” button gives a person too little information to recognize a manipulated action. Approval must attach to the specific operation and cannot be substituted by text retrieved from a source.

An answer can transmit data through a generated link or externally loaded image. Review which output formats the interface actually renders. An internal knowledge assistant often has no need for arbitrary external active content. Build citations from known source metadata rather than accepting any destination invented by the model.

Caches need the same access discipline as retrieval. An answer generated for an authorized employee must not be served to another user under an overly broad question key. Account for organization, permissions and source version in the caching design, or avoid shared caching for sensitive answers.

Conversation history can preserve earlier content. After access is revoked, filtering only the next search may not be enough. Decide how earlier context is handled and whether the conversation must restart for the affected collection.

Pattern matching can identify obvious commands. A separate classification model may flag further suspicious passages. Both can reject harmless content and miss new attack variants. Treat their output as one signal among several controls, not a general security certificate.

A blocked source also needs a recovery workflow. Who decides whether it can return? Which version is reviewed? How do you prevent the next scheduled import from reactivating the same blocked version? Without those decisions, filtering may create only a temporary interruption.

The observed search competitor Promptfoo publishes methods for red-teaming RAG applications. Its tooling can support systematic generation and evaluation of your own test cases. Passing a tool-generated evaluation remains evidence about the tested configuration and cases, rather than every possible future input. Promptfoo: RAG red teaming.

When a poisoned source is suspected, block its active version and, where appropriate, connected tools. Preserve necessary technical evidence under suitable access controls: source identity, processing version, affected request and the action actually executed. The entire organization does not need access to a full conversation transcript.

Inspect dependent passages, stored summaries and answer caches. Reimporting a cleaned file does not necessarily remove every previously generated derivative. Identify these relationships during normal operation so that an incident response can follow them.

Before restoring access, reproduce the incident as a controlled regression case. Also check whether the correction unnecessarily blocks legitimate questions. A system that refuses every request no longer fulfills its knowledge-assistance purpose.

Public product documents. For an assistant restricted to public product documents, source integrity and avoidance of invented commitments are central. A synthetic secret is mainly a test fixture; the real failure could be an altered price or support promise. Include attempts to make an unrelated source appear to provide an official guarantee.

Confidential internal material. An internal assistant handling confidential material adds separation between teams. Ask the same question as two users with different rights. Correct answers may differ. A shared index is not inherently unsuitable, but its permission filters must work on every retrieval path.

Write access. An assistant with write access also needs checks connecting the requested task to the proposed operation. Asking about a delivery date does not authorize changing an order. Action validation should consider that relationship rather than merely checking whether the requested function exists.

These deployment profiles require different priorities and evaluation cases. Record the selected profile and explicitly excluded capabilities. If new tools are enabled later, the record makes it clear that the threat model changed and another acceptance review is necessary. A small capability switch may matter more than a substantial redesign of the chat interface.

Agree on observable release conditions for a RAG assistant:

• restricted sources never reach model context
• unauthorized tool actions fail technically
• manipulated citation destinations cannot cause uncontrolled data transfer
• withdrawn content disappears from the active index and relevant caches
• security alerts have an accountable recipient

Evaluation scope. Record the limits of the evaluation. Which source types, roles and tools were covered, and which were excluded? A read-only bot evaluation does not validate an email-sending agent added later. Repeat the relevant cases when adding sources, expanding permissions, changing models or altering how answers are displayed.

Operational failure cases. Include operational failure cases as well as deliberate attacks. If identity resolution is unavailable, the assistant should not assume broad access. If the source approval service fails, a new document should not silently enter the trusted collection. The correct fallback depends on the application, but it must preserve the intended boundary.

Usability and security evidence. Finally, keep usability and security evidence together. Record legitimate questions blocked by controls alongside attacks that succeeded. This makes trade-offs visible and helps owners improve the implementation without removing essential restrictions merely because a single demonstration looked inconvenient. Security then becomes an inspectable operating property rather than a sentence in a system prompt.

These modernagents guides cover related decisions about AI knowledge assistants:

• AI knowledge assistants: sources, access and answers
• Preserve SharePoint permissions in RAG answers
• Diagnose wrong RAG answers and useful abstention
• Secure chatbot identity checks and business actions

For a workflow review, start with the task, systems involved and typical exceptions. Discuss it with modernagents.