Diagnose wrong RAG answers and useful abstention

An invented company policy is an obvious error. More subtle failures combine individually plausible fragments. A number comes from one document, a deadline from another, and an approver's name from the model's general knowledge. Together they form a rule that no approved source actually states.

A genuine link can also mislead. The linked policy may discuss the subject without supporting the specific claim. A correct passage may apply to Germany while the question concerns an Austrian office. An old version can be the right evidence for a historical question and the wrong evidence for a decision today.

Microsoft's RAG evaluation guidance separates dimensions including groundedness and relevance. This supports an essential distinction: an answer can faithfully reflect the supplied context while failing to answer the user's question. Microsoft: end-to-end evaluation.

The following policies are fictional and are not travel or tax guidance. An internal policy raises the hotel allowance to €120 from September. The earlier version specifies €95. An appendix requires separate approval for stays during trade fairs.

Someone asks: “Can I book a €115 room during the trade fair next week?” The assistant answers yes and cites the current €120 allowance. The number is correct, but the omitted exception changes the decision. The error is an unsupported permission, rather than a fabricated amount.

A better answer gives the general allowance, explains the trade-fair exception and identifies the approval route. If the appendix is unavailable, the assistant should not issue a booking approval. It can explain the missing information and identify who can resolve it. Evaluation must therefore consider the action the answer encourages, not merely whether individual words match a reference answer.

Break representative answers into independently checkable claims. This is a proposed review method, not the result of a completed benchmark.

Evidence should lead to the relevant passage. Retain document identity, version, section and page where applicable alongside the extracted text. A citation that opens only a portal homepage leaves the employee to repeat the search. This increases verification effort even when the answer happens to be correct.

Diagnose a wrong RAG answer stage by stage, in this order:

1. Source. First establish whether approved information exists. Retrieval cannot reliably create missing organizational knowledge.
2. Extraction. Next inspect extraction: did the importer preserve the document's meaning? Tables often lose column headings or footnotes that are essential to interpretation.
3. Search. Then inspect search. Was the necessary passage among the retrieved candidates?
4. Reranking and context assembly. If so, was it removed by reranking or context assembly?
5. Answer generation. If the complete evidence reached the language model and the answer was still wrong, focus on answer instructions, model behavior or downstream checking.

This sequence prevents unnecessary infrastructure work. Switching to another vector database will not restore a missing footnote. A larger context window does not resolve an incorrect effective date. Stricter wording does not replace technically enforced access permissions.

Define distinct outcomes. “No relevant source found” differs from “Two applicable sources disagree.” The following table pairs three situations with the response a RAG assistant should give.

Your organization may decide that an approved policy takes precedence over a team wiki, but that precedence needs an accountable business owner.

A confidence number generated by the model is not a sufficient substitute. Use observable conditions instead of a model-generated confidence number:

• required metadata is present
• an applicable source was retrieved
• material exceptions were included
• no higher-priority source contradicts the proposed conclusion

These conditions can be inspected and tested.

A source may contain instructions aimed at the assistant as well as ordinary business information. Treat those instructions as source data. A paragraph asking the model to disclose other files does not become an authorized command merely because it entered the knowledge base.

OWASP identifies RAG poisoning and indirect prompt injection as distinct attack routes. Source admission, access enforcement and restricted tool permissions therefore belong outside the wording of the final answer. OWASP: RAG security. See the prompt injection guide for the implementation implications.

Collect real work questions and add deliberately difficult variants. Include current and historical questions, missing details, conflicting evidence, knowledge that does not exist and questions involving restricted access. Include German and English wording if both are used by employees.

Score retrieval of the necessary passage separately from support for the answer's main claims, preservation of exceptions and usefulness of the next step. A reviewer should be able to judge the answer without first seeing a second model's score. Automated evaluators can help prioritize a large collection, but they do not establish which business decision is acceptable in a critical case.

Rerun this collection after changes to source content, chunking, retrieval or the model. Add new production failures as they emerge. Sensitive conversations should not automatically become permanent entries in a broadly shared test spreadsheet; create controlled or anonymized cases where appropriate.

Name the authoritative sources and their precedence. Decide when the assistant answers, asks a follow-up question or hands the case to a person. Set a measurable error tolerance appropriate to the task. An HR decision needs a different approval process from locating cafeteria opening hours.

Passing a test establishes behavior on the tested cases. It does not guarantee future correctness. The essential operating loop is to capture an error report, preserve the relevant source and answer versions, diagnose the cause, correct it and retain the case in regression testing. The RAG acceptance plan helps make those decisions explicit before a wider rollout.

These modernagents guides cover related decisions about AI knowledge assistants:

• AI knowledge assistants: sources, access and answers
• Test a RAG knowledge assistant: sources, access and answers
• Prepare PDFs, scans and tables for RAG
• Keep RAG knowledge current through updates and deletion

For a workflow review, start with the task, systems involved and typical exceptions. Discuss it with modernagents.