Keep RAG knowledge current through updates and deletion

Three timestamps describe a document in a RAG collection, and their values can differ substantially.

A policy published in August may become effective in October. A document imported again in September may not have undergone a substantive review for years. Access revoked yesterday may need to disappear immediately even though the document text has not changed. One timestamp cannot represent all forms of freshness.

Keep source identity, version, effective-from and effective-to dates where relevant, import status and the time of the latest permission check. An explicit unknown value is preferable to inventing a date for missing metadata.

The following table lists the events a RAG update pipeline must recognize with the required response to each.

Source interfaces provide different mechanisms. Microsoft Graph offers delta queries for drive items to track changes, including deleted items. This is a synchronization mechanism, not proof that a RAG application has correctly removed its own copies. The application still has to apply those events to its search and answer stores. Microsoft Graph: driveItem delta.

A new document version in a RAG collection is activated in these steps:

1. Process a new document version outside the active search collection first.
2. Validate extraction, required metadata and source identity.
3. Then activate it for the intended audience and applicable period.
4. Depending on the business purpose, retain the previous version for historical questions or block it entirely.

Every passage needs a shared parent document identifier. Otherwise deleting the main record can leave orphaned text fragments discoverable. Also record the processing version: changing the extraction method is a different event from changing the source document.

Activation should be complete or visibly incomplete. If a hundred-page manual was only partially imported, the application must not quietly present the result as a complete knowledge source. The missing page may contain the exception that changes an answer.

Suppose a fictional service policy lists an incorrect contact address. The following fictional timeline shows how the correction reaches the assistant.

The update must therefore invalidate dependent saved answers as well as replace searchable passages. A control question about the contact address then verifies the answer users actually see. A successful import status is insufficient evidence of a successful correction.

If the old file is also withdrawn, inspect direct source links and exported conversation views. Information already sent to people cannot simply be recalled by changing an index. Critical corrections need an organizational notification route as well as a technical update path.

Some systems automatically detect new or modified files but need additional configuration to recognize removals. Azure documents separate change and deletion detection mechanisms for Blob indexers. Configure a deletion strategy appropriate to the source before relying on it. Azure AI Search: changed and deleted blobs.

Test a deletion along the full route from source to answer:

1. remove the file
2. run synchronization
3. search for an exact passage
4. ask a paraphrased question
5. open an old citation

Repeat the check in a conversation that was already open. Its history may contain information that should not be reused in a fresh answer.

Define separate retention rules for active search, operational logs and backups. A backup is not an active search collection, but it still needs a retention policy and a restoration process that does not accidentally republish withdrawn material.

Expired credentials, API limits and changed file layouts can stop ingestion. Monitor when each source was last processed successfully and completely, rather than merely when a scheduled job ran. A healthy scheduler can coexist with a knowledge collection that has not changed for weeks.

A clearly labelled older version may be acceptable for low-risk information. For time-sensitive rules or revoked access, excluding the affected source may be necessary. Make this an agreed operating decision instead of leaving the model to improvise a fallback.

Recovery can deliver events more than once. Stable identities and version comparisons help repeated processing avoid duplicate search records. If a change cursor is no longer valid, plan a complete reconciliation rather than silently continuing with the next newly created file.

Successfully synchronizing a source does not show that its information is correct. An unchanged page may already be obsolete. Set review intervals according to the topic and identify someone who resolves reported contradictions. A building map and a frequently changing product approval require different review schedules.

Atlassian, for example, documents explicit content ownership and ownership transfer. Such features support accountability but do not establish that someone actually reviewed the underlying information. Atlassian: transfer content ownership.

When review is overdue, decide whether the content remains usable with a warning or leaves the answerable collection. A uniform expiry rule can create unnecessary gaps; a rule with no consequences can become a decorative date. The source owner should make that choice according to the impact of a wrong answer.

Verify that RAG updates and deletions have propagated with this checklist:

1. Include a new document, a correction, a deletion, a revoked permission and a future-effective version in the evaluation set.
2. Record the source event time and the time at which user-visible behavior becomes correct. This produces a measurable propagation delay instead of an untestable claim such as “near real time.”
3. Run the same cases after restoring a backup.
4. Document who decides what to do when the agreed delay is exceeded and how to block an affected source.

The RAG architecture should explicitly represent these states, while the acceptance plan connects them to representative business questions.

These modernagents guides cover related decisions about AI knowledge assistants:

• AI knowledge assistants: sources, access and answers
• Build an AI knowledge base with owners and current sources
• Preserve SharePoint permissions in RAG answers
• Diagnose wrong RAG answers and useful abstention

For a workflow review, start with the task, systems involved and typical exceptions. Discuss it with modernagents.