Incremental RAG indexing: keep updates and deletions consistent

An index is a continuously maintained view

Incremental indexing processes detected changes rather than rebuilding everything. Detecting an update is different from detecting deletion. Microsoft documents source-specific change and deletion policies; a successful indexing run does not by itself prove that deleted content disappeared from search.

An editorial document-state design

Store a stable document ID, source version, content fingerprint and processing state. Distinguish source-change time from index-application time to measure freshness lag. When re-chunking, remove obsolete chunks by parent document ID. Propagate permission changes even when document text remains unchanged, including relevant caches.

Make deletion and moves explicit

Where soft deletion is used, retain deletion markers long enough for synchronization to observe them. Define whether moves preserve identity; file paths alone are not universally stable identifiers. Reconcile source and index inventories periodically to find orphaned documents. Removing an item from search is distinct from deleting backups or audit records.

For more on this topic, read Connecting AI answers to organizational documents in the Liyan knowledge base.

Exercise the complete lifecycle

Create a test document, retrieve it, update it, change its permissions and delete it. Check search, citations and caches after each step. Measure actual synchronization delay rather than scheduler execution time. Expose failed files and bounded retry states so partial failures do not masquerade as complete synchronization.

Exercise the complete document lifecycle

Create a synthetic HR policy, update one clause, rename it, revoke a group's access and finally delete it. Measure each change independently. Confirm that old answers cannot return from either the index or response caches. Distinguish archival from deletion according to source semantics; successful initial ingestion proves little about later lifecycle behavior.

Keep a per-document synchronization ledger

Track source ID, content version or fingerprint, permission state, last observation, job ID and processing outcome. Distinguish publication time from ingestion time. Mark a document searchable only after its chunks are validated. Per-document states expose failures hidden by a successful overall job. Preserve the mapping from each published version to its complete chunk set.

Handle interruption and reconcile independently

Replacing matching chunk numbers is insufficient when the new version has fewer chunks. Publish a consistent version and remove its predecessors according to your index architecture. Interrupt processing during construction and publication to test recovery. Keep permission enforcement effective while content processing lags. Monitor queue age, repeat failures and source-index differences; verify source connectivity before interpreting missing entries as deletions. Agree separate propagation targets for edits, revocations and deletion, and provide targeted reprocessing plus periodic full reconciliation.

A recovery drill for changes that arrive mid-job

Start with a searchable document, then replace it with a shorter version that produces fewer chunks. Check that extra chunks from the predecessor disappear. Rename the document without changing the body and verify both result labels and source links. Revoke a test group's access while content processing is queued. These exercises expose different paths that a simple upload test does not cover.

Stop the worker during new-version construction and again near publication. On resumption, inspect whether readers see one coherent version and whether the synchronization ledger identifies unfinished work. The exact publication mechanism depends on the index architecture; the acceptance requirement is a known, recoverable state rather than a mixture of old and new fragments. Permission controls need their own timely enforcement so a content-processing backlog does not leave obsolete access active.

Finally, simulate an unavailable source listing. A sudden drop in discovered documents should prompt a connectivity and completeness check before any broad deletion. Compare the source and index periodically to detect missed events, and provide document-specific replay for repair. Report the age of the oldest unapplied change, repeated failures and unresolved discrepancies. Agree different propagation targets where creation, ordinary editing and access revocation have different consequences. The operational owner should be able to explain why a particular version is searchable and how to repair it, rather than relying on a green status for the overall scheduled job.