Self-Hosted vs Managed LLM Deployment: Which Fits Your Enterprise?

October 14, 2026 · LuMay AI

How you run a large language model (LLM) shapes your security, cost, and speed for years. The real question is who controls inference, who handles incidents, and who owns ongoing operations.

There is no universally cheaper or safer option. The right answer depends on workload, data sensitivity, latency targets, and your team's ability to run GPU infrastructure. This guide separates the options, compares them, and shows how to test them before committing.

Quick Recommendation

• Start with a managed model API for fast launch, variable demand, and small platform teams, as long as your data rules allow a provider.
• Choose a dedicated managed endpoint when you want a chosen model and predictable capacity without running the serving infrastructure.
• Choose self-managed serving for strict isolation or air-gapped needs, deep model control, or steady high volume, and only if you can staff operations.
• Choose hybrid when workloads differ in sensitivity, volume, or latency.
• Do not treat privacy, speed, or savings as automatic. They come from architecture, workload, and contract, not from the label.
• Measure cost per successful task, not token price, and decide with a pilot, not a slide deck.

Sensible default: Pilot a managed option unless a documented constraint rules it out. Switch only when quality, risk, or cost evidence supports it.

Plain-English Definitions

• Self-managed model serving: You run model weights on infrastructure you control, on premises or in your cloud account. Tools such as vLLM's online server expose an OpenAI-compatible API with health and metrics endpoints, but you still own capacity, scaling, patching, access control, and recovery.
• Managed model API: A provider runs shared infrastructure and usually bills per token. You call it over the network.
• Dedicated managed endpoint: A provider runs a model you select on capacity set aside for you. Hugging Face Inference Endpoints is an example: autoscaling, logs and metrics, support for engines such as vLLM, and no Kubernetes or CUDA management. Dedicated does not mean on premises or customer-owned hardware.
• Hybrid: A router sends each task to the right backend, for example sensitive extraction to a private model and general drafting to an API.

Two separations matter:

• Location is not responsibility. A model can run in your cloud account while a partner operates it.
• Application hosting is not inference hosting. A privately hosted application can still send prompts to an external model. Check every model call, tool, log, and fallback path.

Who Manages Each Layer?

Licensing and Shared Responsibility

• Open weights are not automatically unrestricted. Check commercial-use, redistribution, size or revenue limits, and acceptable-use terms before you build.
• Model control means pinning versions, fine-tuning, and auditing behavior. Managed APIs may change or retire models on the provider's schedule.
• Shared responsibility applies everywhere. AWS's Amazon Bedrock guidance says AWS protects the infrastructure while you remain responsible for controlling your content. It also states that model providers have no access to Bedrock logs or customer prompts and completions.

Security Duties You Cannot Outsource

• Access control: Decide who may call the model and which data each role may send.
• Data classification: Decide what may leave your boundary before you pick a provider.
• Log protection: Prompts and outputs can hold sensitive data, so secure logs like production databases.
• Configuration: Encryption, private networking, and least-privilege roles remain your settings. The AWS page requires TLS 1.2, recommends TLS 1.3, and advises multi-factor authentication, least-privilege access, and activity logging.

Comparison at a Glance

These are typical tendencies. Verify each vendor's terms.

When Each Approach Fits

Self-managed fits when

• Isolation is mandatory: air-gapped or tightly regulated environments where data must not leave your network.
• You need model control: fine-tuned, pinned, or specialized open-weight models with custom runtimes.
• Volume is steady and high, so reserved GPUs stay busy.
• You have the skills: GPU operations, patching, security, on-call engineering, and disaster recovery.
• Watch for: idle hardware, upgrade regressions, and the false assumption that private means compliant.

Managed API fits when

• Speed matters: a pilot must ship within weeks.
• Demand is spiky or unproven, so you avoid paying for idle GPUs.
• Frontier models are required and no open-weight model meets your quality bar.
• Your team is small and should focus on the application.
• Watch for: token cost growth, provider model changes, quotas, and regional limits.

Dedicated endpoint fits when

• You want a selected model with clear capacity boundaries but not full infrastructure ownership.
• Watch for: paying for reserved capacity that sits idle.

Hybrid fits when

• Sensitivity differs by task: restricted data stays inside your boundary while general tasks use a managed API.
• Cost and quality differ by task: smaller models handle simple work, stronger models handle complex reasoning.

Four habits keep hybrid manageable:

• Classify and route: Deny unapproved routes by default.
• Fail over safely: Fall back to a second provider or smaller model without breaking residency rules.
• Keep one interface: Use a standard API or gateway so applications do not depend on one backend.
• Evaluate every route: Track quality, cost, and audit trails separately for each model path.

What Full Lifecycle Cost Includes

Token price is only one line item. Compare the full cost of running a reliable workload at equal quality.

• Compute and utilization: Idle GPUs still cost money. Utilization drives self-hosted economics.
• Storage: Model weights, container images, logs, vector indexes, and backups.
• Networking: Private connectivity, data transfer, and load balancing.
• Engineering: Setup, upgrades, tuning, testing, and on-call time.
• Security and governance: Access controls, audits, vendor assessments, and vulnerability management.
• Redundancy: Spare capacity for failures and traffic peaks.
• Observability: Traces, metrics, logs, and evaluation runs.
• Provider charges: Tokens, endpoint hours, commitments, support tiers, and overages.
• Transition: Migration, licenses, and eventual exit costs.

A useful rule: managed options shift operational risk to a provider; self-hosting shifts it to you. Neither is inherently cheaper.

Illustrative TCO Calculation

Every number below is a hypothetical placeholder in "cost units" (CU), not a market price. Replace each with your own quotes and measurements.

Equivalent workload: 10 million tasks per month, 2,000 input and 500 output tokens each, peak concurrency of 40. All options serve the same quality class, validated on one evaluation set, with an assumed 95% success rate.

Formulas

• Managed API: (input tokens in millions x input rate + output tokens in millions x output rate) x (1 + retry rate) + overhead.
• Nodes needed: round up (peak concurrency / concurrency per node) + spare nodes.
• Self-managed: nodes x monthly node cost + engineering + security + observability and network.
• Dedicated endpoint: nodes x monthly node cost x (1 + provider premium) + its smaller engineering, security, and observability costs.
• Cost per successful task: monthly cost / (tasks x success rate).

Assumptions

• Managed API: 1 CU per million input tokens, 4 CU per million output tokens, 5% retries, 3,000 CU overhead.
• Capacity: 20 concurrent requests per node (measure this yourself), 1 spare node, 14,600 CU per node per month.
• Self-managed fixed costs: engineering 6,000 CU, security 2,000 CU, observability and network 1,500 CU.
• Dedicated endpoint: 20% provider premium on node cost; engineering 2,000 CU, security 1,500 CU, observability and network 1,400 CU.
• Fixed costs stay the same when volume grows.

Lesson: Managed cost grows with tokens. Self-managed cost grows in steps as you add nodes, so high, steady utilization can reverse the ranking. Your real throughput, prices, staffing, and quality results may change it again, so find the break-even point only after you measure them.

Three Hypothetical Scenarios

These examples are invented for illustration.

Scenario 1: A retailer's support assistant

• Situation: Seasonal traffic spikes, modest data sensitivity, small platform team.
• Recommendation: Managed API, with rate limits, spend alerts, and a tested fallback.
• Trade-off: Faster launch and no idle GPUs, in exchange for provider dependence and regular review of usage charges and data terms.

Scenario 2: A regulated lender's document analysis

• Situation: Strict residency rules and steady, high volume.
• Recommendation: Start with a dedicated endpoint in a controlled network to prove quality, then move stable high-volume work to self-managed serving if utilization justifies it.
• Trade-off: Stronger control over data paths and model versions, but higher staffing and redundancy cost.

Scenario 3: A multi-country insurance group

• Situation: Different residency rules by jurisdiction; some tasks are sensitive, others are general drafting.
• Recommendation: Hybrid routing, with regional private or dedicated endpoints for sensitive work and a managed API for lower-risk tasks.
• Trade-off: Each workflow follows its own rules, but routing, audit consistency, and failover add governance work.

Decision Checklist

• Data: Which prompts, documents, outputs, logs, and backups may leave your boundary?
• Quality: Does an open-weight model pass your evaluation set, or do you need a frontier model?
• Load: What are average and peak requests, concurrency, and token lengths?
• Latency: What are your p50, p95, and p99 targets under realistic load?
• Model changes: Can you accept provider updates, or do you need version pinning?
• Availability: What recovery time and failover behavior do you need?
• Operations: Who responds at 2 a.m., patches dependencies, and tests recovery?
• Contracts: What do terms say about retention, training use, regions, model retirement, and exit?
• Utilization: Are you comparing a fully used self-hosted cluster against a lightly used managed option?
• Economics: What is the cost per successfully completed task over 12 months?

Four-Week Pilot Plan

Run two or three candidates against the same workload, prompts, retrieval, and evaluation set.

1. Week 1: Set the ground rules. Classify data, pick representative tasks with real input and output lengths, and set pass criteria: quality floor, latency target, cost ceiling, and maximum recovery time.
2. Week 2: Measure quality. Test identical prompts and retrieval. Use human-reviewed samples; treat automated scores as signals.
3. Week 3: Measure speed and cost. Load-test p50 and p95 latency, throughput, and utilization at expected concurrency. Compute cost per successful task using the formulas above.
4. Week 4: Test recovery. Simulate rate limits, node failures, a bad release, and fallback routing, and time restoration. Review results with security and business owners.

Decision rule: Choose the least operationally complex option that meets your quality, risk, performance, and cost targets. Re-test quarterly.

Why Choose LuMay

LuMay's deployment page documents five models: LuMay SaaS, customer cloud (bring your own cloud), private cloud or VPC, on-premises Kubernetes with full air-gap capability, and hybrid. LuMay ships as Docker containers, and the same images run in every deployment model.

• Model strategy: LuMay's LLM development services compare commercial and open-source models against quality, cost, latency, and control goals, and include private open-source model deployment and fine-tuning.
• Production operations: The same page lists model routing, fallback behavior, rollback procedures, and latency, availability, and cost tracking. Managed AI services cover ongoing monitoring and optimization.
• Governance: The security page lists SSO, role-based access, database-level tenant isolation, PII controls, human approval gates, and audit trails, and the site lists SOC 2 Type II and ISO 27001 certification.
• Breadth: The services hub spans integration, automation, RAG, and agents.

The deployment page does not say where model inference runs in each option, so confirm that during discovery. Contact LuMay to scope a deployment pilot.