A RAG assistant can retrieve the right passage from a document that should no longer exist. Good retrieval depends on keeping the index synchronized with its sources, including updates, deletions and permission changes.
Incremental indexing processes changes instead of rebuilding the entire corpus each time. It can reduce repeated work, but it also creates synchronization responsibilities. These seven practices help make the process recoverable and auditable.
TL;DR
- Give each source document a stable identity independent of its filename.
- Detect content and permission changes separately.
- Make processing repeatable so retries do not create duplicate chunks.
- Treat source deletion as a dedicated event and reconciliation problem.
- Publish complete document versions together, and measure how long changes take to reach retrieval.
Which update strategy fits
| Strategy | Useful when | Main limitation |
|---|---|---|
| Full rebuild | Corpus is small or a clean reset is needed | Repeats extraction and embedding work |
| Event driven updates | Source supplies dependable change events | Lost events can cause drift |
| Scheduled incremental scan | Source exposes a change cursor or modified state | Updates wait for the next scan |
| Incremental processing plus reconciliation | Correctness needs an independent completeness check | More operational components |
1 Assign stable source and chunk identities
Use the source system's durable document identifier where available. Record the tenant, collection, document ID and version. A filename or path can change without creating a new logical document.
Derive chunk identity from the source identity, version and chunking configuration. Keep enough metadata to find every chunk belonging to a document. LangChain's RecordManager reference describes tracking document hashes, timestamps and source identifiers.
Pros: Updates and removals become traceable. Cons: Identity mapping must survive file moves and connector migrations.
2 Detect content changes deliberately
Compare a source version, content hash or reliable change token before reprocessing. A metadata edit may not require new embeddings, while an extraction or chunking change may require reprocessing unchanged source bytes.
Keep the extractor, chunker and embedding model versions in the processing record. Use the RAG chunking guide when deciding which changes alter chunk boundaries.
Pros: Avoids unnecessary embedding work. Cons: Weak change signals can miss updates, and hashing large source files also has a cost.
3 Make ingestion idempotent
A retry should converge on the same current index state. Use deterministic identifiers, upsert semantics where supported and checkpoints that distinguish extraction, embedding and publication.
For example, if a worker crashes after embedding but before acknowledging the event, replay should not create another copy of the same chunk. Keep a durable failure record and a retry policy for temporary errors. Avoid acknowledging success before the index write is complete.
Pros: Recovery is safer and duplicate growth is reduced. Cons: The team must design persistence and retry boundaries explicitly.
4 Handle deletions separately from updates
Process deletion events or tombstones and remove all associated chunks, cached answers and dependent metadata. Also reconcile against the authoritative source inventory so a missed event does not leave content searchable indefinitely.
LangChain's indexing API documents cleanup modes with different scopes. Incremental cleanup for source IDs seen during a run must not be assumed to discover every source document deleted outside that run. Choose cleanup behavior deliberately and test its exact scope.
Pros: Removed content stops influencing answers. Cons: Deletion must propagate to more than the vector store, including caches and derived views.
5 Synchronize permissions as their own change stream
A document can remain byte-identical while access is revoked. Update authorization metadata independently of embedding work and enforce the permission boundary during retrieval.
Decide how queries behave while an access update is delayed. For sensitive content, a live authorization check or temporary suppression may be appropriate. The SharePoint permission-aware retrieval guide explains related access questions.
Pros: Access changes do not wait for a full re-embedding cycle. Cons: Group membership, inherited permissions and sharing links can complicate the effective access model.
6 Publish complete document versions
Stage the new chunks and metadata, validate them, then make the new version visible using a mechanism supported by the storage architecture. Avoid a query receiving half the old document and half the new one.
If the database does not support an atomic multi-record change, use an active version pointer, generation filter or another explicit publication strategy. Retire the previous version after the transition succeeds.
Pros: Readers see a coherent version. Cons: Staging can temporarily require more storage and needs a rollback plan.
7 Reconcile and measure freshness
Measure source change time, detection time, processing completion and query visibility. Track backlog age, failures, unexpected document counts and orphaned chunks. An empty queue does not prove that no source events were lost.
Periodically compare the source inventory with indexed document identities and versions. LangChain's source synchronization article provides background on avoiding repeated indexing work; the broader operating controls still depend on your source and storage systems.
Pros: Reconciliation finds silent drift. Cons: Scans consume resources and may need scheduling around source API limits.
A worked update sequence
Suppose a policy document moves from version 3 to version 4. The connector detects the change, records its identity, extracts the new text and creates a staged set of chunks. Validation checks the chunk count, source links and access metadata before version 4 becomes active.
A retry after a partial write resumes using the same identities. If the source later deletes the document, a deletion event removes it from retrieval and invalidates dependent caches. A scheduled inventory check independently confirms that no chunks remain. The version numbers here are illustrative.
Acceptance tests
| Scenario | Evidence to collect |
|---|---|
| Same event delivered twice | One current logical version |
| File renamed without content change | Identity preserved without duplication |
| Worker stops during publication | Controlled retry or rollback |
| Source deleted without a delivered event | Reconciliation finds and removes it |
| Access revoked without content change | Unauthorized retrieval fails |
| Chunking configuration changes | Affected documents are reprocessed |
Benefits and operating costs
Incremental processing can reduce repeated work and shorten update delays. Its cost is managing state, connector behavior, publication and recovery. A small corpus may still be simpler to rebuild. Choose based on change rate and freshness needs, not the assumption that incremental is always cheaper.
Frequently asked questions
Is a modified timestamp sufficient
Only when it reliably represents every change you need to detect. Some permission or extraction changes need separate signals.
Does changing the embedding model require a rebuild
Usually the affected corpus needs new embeddings and a controlled migration. Do not mix incompatible vectors in one search path without an explicit design.
Does incremental cleanup remove every deleted source
Not necessarily. Cleanup scope varies by implementation. Explicit deletion handling and reconciliation are needed when absent documents must be discovered.
How often should reconciliation run
Choose a cadence from the acceptable stale-content window, source limits and corpus size. Verify that the process completes within that window.
Keep freshness in the architecture
Test synchronization alongside answer quality. Use RAG evaluation to measure the resulting behavior and LuMay AI engineering services to discuss connector and indexing requirements.
Sources and further reading
- RecordManager | langchain_core | LangChain Reference
- index | langchain_core | LangChain Reference
- Syncing data sources to vector stores
Sources and linked destinations checked on 1 November 2026. Product capabilities and documentation can change.
