Custom RAG vs Managed RAG: Which Delivery Model Fits Your Enterprise?

The short answer. Custom RAG means your team builds and runs every layer of the retrieval pipeline: parsing, chunking, embeddings, vector storage, ranking, and access control. Managed RAG hands most of those layers to a cloud service such as Amazon Bedrock Knowledge Bases, Azure AI Search, or Google's RAG Engine. Custom buys control and portability; managed buys speed and lower operational load. Most large enterprises end up with a hybrid of the two.

A proof of concept can make either approach look easy. Production asks harder questions: Who is allowed to retrieve a document? How quickly do updates and deletions appear? What happens when the search finds nothing useful? Choose a delivery model by testing those requirements on your own data, not by assuming that "managed" means secure or that "custom" means better.

1. What custom and managed RAG mean

Retrieval-augmented generation (RAG) grounds a large language model's answers in content retrieved from your own documents and data. That keeps answers current and traceable without retraining the model whenever a policy changes. If you are still weighing RAG against model training, start with RAG vs fine-tuning.

The delivery question is different: who builds, runs, and is accountable for each stage of the retrieval pipeline? In practice there are four delivery models, not two.

• Fully managed RAG. The provider runs ingestion, parsing, chunking, embedding, storage, and retrieval behind an API.

• Managed retrieval components. You assemble the pipeline from managed building blocks, such as a managed search index or vector store, and write the orchestration yourself.

• Custom RAG. Your engineers select and operate every component, from parsers to the vector database, on infrastructure you control. It can still run on a public cloud.

• Hybrid RAG. You use managed services for commodity layers and custom code where retrieval quality, security, or portability is a differentiator.

These labels are not standardized procurement categories. A managed vector database is not the same as a fully managed RAG workflow, and a customer-managed knowledge base is not necessarily a fully custom application. The major clouds already split their own offerings along this line (as documented in October 2026): AWS offers a managed and a customer-managed knowledge base in Amazon Bedrock; Microsoft offers both agentic retrieval and a classic RAG pattern in Azure AI Search; and Google's RAG Engine, now documented under the Gemini Enterprise Agent Platform, defaults to a managed vector database but lets you bring your own.

2. How both architectures work

Both models run the same pipeline. The difference is where your responsibility starts and stops.

1. Connect sources such as SharePoint, wikis, databases, and APIs.

2. Parse and clean text, tables, and images, including OCR.

3. Chunk and enrich content with metadata such as owner, date, and access-control lists.

4. Embed and index chunks in vector and keyword indexes.

5. Retrieve and rank with hybrid search, permission filters, and reranking.

6. Generate answers with citations to sources.

7. Evaluate and monitor recall, groundedness, latency, and cost.

In a fully managed service, steps 2 to 5 run inside the provider's platform. AWS states, for example, that a Bedrock managed knowledge base handles ingestion, storage, indexing, and retrieval, with service-managed embedding and reranking by default. With managed components, you own the orchestration: Azure describes its classic RAG pattern as your application sending a query to Azure AI Search and passing the results to an LLM separately. In a custom build, every step is your code and your infrastructure. Hybrid designs mix these per data source.

Retrieval is one part of a wider system. LuMay's enterprise AI architecture patterns explains how it fits alongside gateways, agent orchestration, and governance.

3. Infrastructure and engineering requirements

Managed RAG shrinks the infrastructure you run, not the expertise you need.

What a managed deployment still requires

• A cloud landing zone with identity, networking, logging, and key management.

• Service roles granting access to sources and vector stores.

• An application team, plus a data owner who keeps content current.

What a custom build adds

• Data engineering: connectors, parsing, OCR, and incremental sync.

• Search engineering: embeddings, chunking, hybrid search, and reranking.

• Platform operations: vector database scaling, backups, and on-call support.

• Security engineering: permission sync and leak testing.

• LLMOps: evaluation, regression tests, and model upgrades.

Recommendation: Before choosing custom, name the owner for each of those five areas, and name who will still maintain the vector database in two years. If two or more areas are unstaffed, start managed or hybrid and close the gaps first.

4. Retrieval quality, customization, and scalability

Managed services have closed much of the retrieval-quality gap. The remaining difference is how far you can tune the pipeline for your own content.

What managed services now include

• Bedrock's managed knowledge base offers agentic retrieval that breaks complex queries into sub-queries.

• Azure AI Search agentic retrieval runs sub-queries in parallel and applies semantic reranking to each. Microsoft notes it adds latency compared with a single-query pipeline, and LLM-based query planning is still in preview.

• Google's RAG Engine documents reranking, metadata filtering, and cross-corpus retrieval.

Where managed customization stops

Managed services expose options, not internals. On a Bedrock managed knowledge base, embedding choices are limited to supported Bedrock models and chunking is built-in or fixed-size. Custom pipelines can add clause-level chunking for contracts, table-aware parsing for financial filings, domain-tuned embeddings, graph-based retrieval, and query rewriting tuned to internal vocabulary. The benefit is control; the cost is that you must validate every improvement.

Scalability

Managed services scale within published quotas and tier limits, which vary by service and region. Custom systems scale as far as you engineer them, but sharding, replication, and re-embedding at scale become your problem. In both models, check connectors, model quotas, and the application tier, which can bottleneck before the index does.

Recommendation: Most quality problems come from data preparation and missing evaluation, not the delivery model. Run a bake-off on 50 to 100 real questions, including abbreviations, conflicting policy versions, long tables, and questions whose right answer is "I don't know", before concluding you need custom retrieval.

5. Security, compliance, governance, and data residency

This is where delivery-model decisions are usually won or lost. The test is not whether a provider is certified; it is whether the specific feature you need supports your controls. Managed services run under a shared-responsibility model: the provider secures the service, while you remain responsible for access mappings, lawful processing, prompt content, and incident response.

Permission-aware retrieval

Users must only retrieve content they are entitled to see, and that check belongs in retrieval, not just the interface. Bedrock's managed knowledge base filters results by document-level access-control lists at retrieval time (except for web-crawled content), and AWS notes document-level permissions are not available on customer-managed knowledge bases. Azure AI Search supports document-level security trimming and can inherit SharePoint permissions and Microsoft Entra ID permission metadata. A citation is not proof that retrieval was authorized.

Encryption, networking, and residency

• Google's RAG Engine supports VPC Service Controls and customer-managed encryption keys, but its documentation states that data residency and Access Transparency controls are not supported. Several regions also require allowlisting.

• Bedrock requires OpenSearch managed clusters used as a knowledge-base vector store to allow public network access; VPC-only domains are not supported.

• "Data residency" applies separately to source files, extracted text, embeddings, prompts, logs, backups, and model requests. An index being in a region does not prove everything else is.

Governance and prompt-injection risk

Whichever model you choose, log the query, the retrieved passages, and the answer for audit, and confirm that deleting a source document removes it from every index. Treat retrieved text as untrusted: the OWASP LLM Top 10 highlights prompt injection and sensitive-information disclosure, and the NIST Generative AI Profile is a useful structure for risk ownership and monitoring. Neither a framework nor a vendor certification makes a deployment compliant by default.

Recommendation: Build a control matrix that lists each requirement (residency, keys, private networking, access trimming, audit) against the exact service feature and region. Custom RAG gives you full control, including on-premises or air-gapped deployment, but you also produce all the audit evidence yourself. See how LuMay approaches AI security and governance.

6. Integration with enterprise systems

A RAG system is only as useful as the sources it can reach, and connector coverage varies more than marketing pages suggest.

• Bedrock's managed knowledge base lists seven native connectors (Amazon S3, SharePoint, Confluence, Web Crawler, Google Drive, OneDrive, and Custom); customer-managed knowledge bases list only S3 and Custom.

• Azure AI Search indexers pull from more than 10 Azure data sources, and agentic retrieval can query remote SharePoint and Bing without indexing them.

• Google's RAG Engine ingests from sources such as local files, Cloud Storage, and Google Drive.

Legacy ERPs, mainframes, industry document-management systems, and CRM records with row-level entitlements rarely have native connectors, and structured data often needs SQL or API retrieval rather than vector search. Static procedures tolerate scheduled sync; inventory, account status, or transactions may need authorized API calls at query time. Keep read-only lookup separate from actions that change records, and use validated transactional APIs with human approval where risk warrants it. Custom connectors should preserve source IDs, versions, owners, and timestamps so every citation traces back to a record.

Recommendation: List every source in scope and mark which ones your preferred cloud connects to natively. If most need custom connectors, you are already building half a custom pipeline. LuMay's AI integration services cover this layer.

7. Deployment speed, maintenance, and vendor lock-in

Speed. Managed RAG reaches a working prototype faster because ingestion, indexing, and retrieval already exist. That head start narrows in production: permission mapping, connector gaps, legal review, and evaluation take similar effort under either model.

Maintenance. Managed services patch and scale themselves, but they also change underneath you. Microsoft notes that agentic retrieval objects created in the Azure or Foundry portals may use preview schemas and need migration to the generally available REST API. Custom systems put every upgrade on your roadmap; changing an embedding model usually means re-embedding and re-indexing the whole corpus.

Lock-in. It lives in three places: the index format, the embedding model, and the orchestration API. Lock-in is not automatically bad; it can be a rational trade for speed. But plan for reversibility.

• Keep raw documents and metadata in storage you control.

• Put retrieval behind an internal API so applications never call the vendor directly.

• Version your evaluation set so you can re-test on another platform quickly.

• Document an exit plan, including re-embedding time and cost.

8. Total cost of ownership and operational trade-offs

Neither model is universally cheaper. Managed RAG converts fixed engineering cost into usage-based spend; custom RAG does the opposite. The most expensive part of enterprise AI is usually human engineering time, not the cloud bill.

How managed RAG is billed

• Azure bills agentic retrieval by tokens: Azure AI Search for retrieval and semantic ranking, and Azure OpenAI separately for query planning.

• Google bills for the RAG Engine-managed Spanner instance when it serves as the vector database in a generally available location.

• AWS states the managed knowledge base includes its built-in embedding model and semantic reranker at no extra cost; generation models are billed separately.

What custom RAG costs

Custom TCO is dominated by people: data, search, platform, and security engineers, plus on-call coverage. Infrastructure adds vector database hosting, compute for parsing and embedding, and GPUs if you self-host models. Hybrid adds cross-vendor network traffic, observability, and failure handling at component boundaries. Both models share hidden costs: content cleanup, evaluation sets, permission sync, and human review of high-impact answers.

Recommendation: Compare cost per accepted, correctly grounded answer over three years, not price per query. Model usage at today's volume, 5x, and 20x. Whether managed or custom costs less depends on your volume and workload, so test it rather than assume it.

9. Comparison: custom vs managed vs hybrid RAG

The table summarizes typical trade-offs. Exact capabilities depend on the specific service, region, and release stage, so verify each row against current documentation.

10. Enterprise use cases and architectural recommendations

These are illustrative scenarios, not client results.

For an architecture tailored to your systems, LuMay's RAG development services describe custom ingestion, retrieval, security, and citation-focused implementation support.

11. Decision framework: custom, managed, or hybrid RAG?

Decide layer by layer, not all at once. Work through these six questions in order; an early hard constraint can settle the rest.

1. Residency: can every in-scope document legally sit in the provider's service, region, and feature set? If not, those sources go custom.

2. Sources: how many have native connectors in your preferred cloud?

3. Quality: does a managed baseline meet your accuracy and citation targets on your own questions?

4. Latency and scale: does p95 latency at peak load fit the experience, within quotas?

5. Team: can you staff ownership of a custom pipeline long term?

6. Exit: could you re-index elsewhere in acceptable time if pricing or terms change?

Rules of thumb

• Choose fully managed when content is standard, sources are covered natively, residency requirements are met, and RAG is an internal utility rather than your product.

• Choose custom when residency, entitlements, or retrieval quality are hard requirements a managed service cannot meet, when documents need bespoke parsing, or when RAG is part of what you sell.

• Choose hybrid when most content is routine but a sensitive or high-value slice needs custom handling. Use managed retrieval for broad internal knowledge and reserve custom builds for specialized, client-facing workflows.

Enterprise RAG decision checklist

• ☐ Every source is classified by sensitivity, residency, and owner.

• ☐ Connector coverage is mapped for each candidate platform.

• ☐ A labeled evaluation set exists, including unanswerable questions.

• ☐ Permission-aware retrieval passes negative tests (zero leaks).

• ☐ Keys, private networking, and audit logging are confirmed per feature and region.

• ☐ Preview versus generally available status is checked for every dependency.

• ☐ Three-year cost per accepted answer is modeled at 1x, 5x, and 20x volume.

• ☐ Owners and an exit plan exist for re-indexing, monitoring, and re-embedding.

For the evaluation step, see 9 LLM evaluation metrics to track before launch.

Conclusion: choose by layer, then prove it

Custom versus managed RAG is not a single build-or-buy vote. It is a series of decisions about who owns parsing, indexing, retrieval, permissions, and evaluation. Start with your hardest constraint, usually residency or entitlements, and let it decide which layers must stay under your control. Use managed services for everything else, behind an internal retrieval API that preserves your exit options.

Then prove the choice with your own documents and questions before scaling, and pick the smallest architecture that passes every non-negotiable gate. If you would like a second opinion, LuMay's RAG development services cover strategy, custom retrieval pipelines, integration, and evaluation, or you can contact the solutions team.

References

Official documentation, checked 13 October 2026. Vendor features change often.

• AWS, Retrieve data and generate AI responses with Amazon Bedrock Knowledge Bases

• Microsoft Learn, Retrieval-augmented generation (RAG) in Azure AI Search

• Google Cloud, RAG Engine overview

• OpenAI, File search tool (hosted retrieval over uploaded files)

• OWASP, Top 10 for LLM applications

• NIST, AI RMF Generative AI Profile