Changing an embedding model changes how your application represents meaning. Updating the query model while leaving documents embedded with the previous model can make retrieval unreliable, even when the vectors have the same length.
An embedding model migration should rebuild compatible document vectors, evaluate retrieval against the current system, and switch query traffic through a controlled release. Keep the previous configuration recoverable until the new path has passed application-level checks.
Why a model change requires more than an endpoint edit
Microsoft's Azure AI Search guidance says indexing and query embeddings should use the same embedding model. Vector length alone does not establish that two models produce compatible representations.
Azure also provides a migration example that creates a target index with updated embeddings. It is a reference implementation with stated limitations, rather than proof that a particular enterprise migration will meet its reliability requirements.
The runbook below is a proposed operating method. Adapt its storage, synchronization, and cutover mechanisms to your search platform.
Create a migration manifest before processing documents
Record what the current retrieval path depends on. Otherwise, a search-quality difference may be blamed on the embedding model when chunking, filters, or reranking also changed.
| Manifest item | What to retain |
|---|---|
| Source snapshot | Document IDs, versions, active status and permissions |
| Transformation | Parser, chunking and preprocessing versions |
| Vector configuration | Model identifier, dimensions and similarity configuration |
| Retrieval configuration | Filters, query transforms, hybrid settings and reranker |
| Release control | Traffic routing, owner and rollback conditions |
Use stable source and chunk identifiers. A filename or array position is insufficient when documents change during the migration.
Build a parallel retrieval path
Keep the existing index available while generating the new vectors. A separate index or an explicitly separated vector field can work, provided the application cannot accidentally compare a query vector with the wrong document vectors.
Version the query encoder and target retrieval configuration together. Treat them as one release unit. Give migration workers bounded concurrency and record failed chunks for resumable processing.
Do not migrate only the text. Carry over tenant boundaries, access metadata, source links, and retirement state. Rebuilding vectors from an old export must not make withdrawn or unauthorized documents discoverable.
Compare results using paired questions
Construct a reviewed query set covering the work the assistant actually performs. Include short factual questions, domain abbreviations, exact identifiers, ambiguous requests, multilingual queries where relevant, and questions with no supported answer.
For each question, define the relevant source set or a review rule. Run the old and new retrieval paths against the same authorized corpus state. Compare relevant-document retrieval, ranking changes, empty-result behavior, latency, and the quality of the final grounded answer.
Inspect regressions by category. A higher overall score can hide worse performance on rare but important queries. Where human review determines relevance, retain the review rationale and resolve disagreements before treating the benchmark as ground truth.
Keep changes synchronized while the rebuild runs
Choose a synchronization strategy explicitly. Options include a short content freeze, a change log replayed into the target, or dual processing into both paths. Each has operational trade-offs.
Track inserts, updates, permission changes, and deletions. Deletion is particularly easy to miss if the backfill scans only documents that currently exist. Use the lifecycle controls in LuMay's guide to retiring RAG content to prevent removed material returning through a migration snapshot.
Illustrative example: two indexes and one release switch
A hypothetical policy assistant has an index called policy-a. The team builds policy-b using a new model, while retaining the same source IDs and initial chunking.
Before switching users, it replays source changes into policy-b, verifies the document manifest, and runs paired retrieval checks under representative permissions. A limited traffic release then uses policy-b and its matching query encoder together.
If the new path starts omitting important policy exceptions, the release controller returns traffic to policy-a. The team verifies that policy-a has received subsequent content and permission changes before treating rollback as safe.
Define cutover and rollback evidence
Agree pass conditions before reviewing the new results. Set thresholds appropriate to the application's risks rather than borrowing a universal accuracy percentage.
- Every expected active source is accounted for, or an exception is approved.
- Permission and deletion tests pass on both ordinary and alternate retrieval routes.
- Important query categories meet the reviewed quality criteria.
- Latency, indexing throughput and operating costs fit the deployment budget.
- The previous configuration can serve current authorized content after rollback.
Include query caches and answer caches in the release plan. Cache entries derived from the older retrieval path should not silently obscure the behavior of the new one.
Should chunking change at the same time?
Usually, isolate the model change first when you need to understand its effect. If new chunking is essential, label it as a combined migration and evaluate the combinations separately where practical. Avoid attributing every improvement to one component.
Keep the migration manifest, comparison results, synchronization record, and cutover decision together. Those records make the next embedding change easier to evaluate and recover.
