An open source LLM gateway gives enterprise teams one controlled entry point between applications and model providers. It can centralize routing, provider credentials, quotas, retries, fallbacks, logging, caching, and observability without forcing every application team to rebuild those controls.
The important caveat is that “open source” does not mean identical feature coverage. Some projects keep most controls in the permissively licensed core; others pair open-source infrastructure with paid identity, governance, clustering, or support features.
This guide compares seven actively maintained gateways specifically as enterprise infrastructure. For the wider system context, LuMay’s 8 Enterprise AI Architecture Patterns places the gateway between applications and model providers to centralize routing, quotas, retries, caching, logging, and policy enforcement.
Key Takeaways
- LiteLLM is the strongest general-purpose default for broad provider abstraction, virtual keys, budgets, retries, and fallbacks.
- Bifrost (Maxim) stands out for open-source virtual-key governance, semantic caching, Prometheus, and OpenTelemetry.
- Portkey Gateway is strong for config-driven routing, retries, fallbacks, guardrails, and lightweight deployment.
- Agent Router fits Kubernetes and Envoy platform teams that want quotas, credentials, routing, and failover at the infrastructure layer.
- agentgateway extends the gateway role across LLM, MCP, and agent-to-agent traffic.
- Apache APISIX and Higress are compelling when AI traffic should share a mature API-gateway control plane.
How We Evaluated the Gateways
Each project was checked against the same enterprise requirements:
- Multi-provider and multi-model routing
- Authentication and API-key controls
- RPM, TPM, token, consumer, team, or budget limits
- Retries, health checks, load balancing, and failover
- Request logging, metrics, traces, and token visibility
- Exact or semantic caching and policy controls
- Docker, Kubernetes, or standalone deployment
- Open-source license and commercial-feature boundaries
A capability is included only where current official documentation or repositories support it.
7 Open Source LLM Gateways Compared
| Gateway | Best For | Routing & Resilience | Auth / Limits | Observability / Cache | License |
|---|---|---|---|---|---|
| LiteLLM | Broad provider abstraction | Load balancing, retries, cooldowns, fallbacks | Virtual keys, budgets, RPM/TPM | Logging integrations, usage/cost, caching | MIT core |
| Portkey Gateway | Config-driven reliability | Conditional routing, retries, fallbacks, load balancing | Key/admin controls; expanding OSS policies | Local logs, caching | MIT |
| Bifrost (Maxim) | OSS governance + observability | Weighted balancing, automatic fallback | Virtual keys, budgets, rate limits | Prometheus, OTLP, semantic cache | Apache 2.0 |
| Agent Router | Kubernetes / Envoy platforms | Provider routing, quotas, failover | Gateway auth, centralized credentials | Envoy/OpenTelemetry ecosystem | Apache 2.0 |
| agentgateway | LLM + MCP + A2A traffic | Weighted/conditional routing, failover | API keys, JWT/OIDC, token/dollar budgets | Metrics, traces, access logs | Apache 2.0 |
| Apache APISIX | Existing API-gateway estates | Multi-instance routing, retries, fallback | API auth plugins + token limits | AI logs, Prometheus, semantic cache | Apache 2.0 |
| Higress | Kubernetes gateway consolidation | Model routing, token failover, fallback | Key/JWT auth, token quotas | AI dashboard, token metrics, cache | Apache 2.0 |
1. LiteLLM
Best for: teams that want the broadest general-purpose model abstraction behind one OpenAI-compatible proxy.
LiteLLM centralizes provider access with model groups, load balancing, retries, fallbacks, virtual keys, budgets, and rate limits. Its router distinguishes retries inside a model group from fallbacks to another group, which makes reliability policy easy to reason about.
The root project is MIT licensed; code under its enterprise/ directory has a separate commercial license. Teams should map SSO and advanced governance requirements before standardizing on the OSS edition.
See the LiteLLM proxy architecture and virtual-key documentation.
2. Portkey Gateway
Best for: teams that want a lightweight gateway with declarative routing and reliability controls.
Portkey’s MIT-licensed gateway supports conditional routing, automatic retries, fallbacks, load balancing, timeouts, guardrails, and caching. It can run locally, in containers, and across edge/serverless environments.
Portkey’s 2.0 line is expanding the open-source feature surface, so buyers should verify the exact stable or pre-release version they intend to deploy before relying on specific governance or caching features.
See the Portkey Gateway repository.
3. Bifrost (Maxim)
Best for: teams that want strong governance and observability features directly in the open-source gateway.
Bifrost documents automatic failover, weighted load balancing, virtual keys, hierarchical budgets, rate limits, semantic caching, built-in request monitoring, Prometheus metrics, OpenTelemetry, and MCP tool filtering as open-source capabilities.
The maximhq/bifrost project is Apache 2.0 licensed. Enterprise additions include clustering, identity-provider integrations, RBAC, and audit logging.
See the Bifrost AI Gateway documentation.
4. Agent Router
Best for: Kubernetes and Envoy teams that want AI traffic policy integrated into existing platform infrastructure.
Agent Router is the new name for Envoy AI Gateway. The project retains the same code, maintainers, CRDs, API group, CLI, and Apache 2.0 license.
It provides an OpenAI-compatible interface, centralized upstream credentials, quotas, provider routing, failover, usage attribution, and Envoy Gateway integration. A standalone aigw run mode is also available for local operation.
See the Agent Router documentation.
5. agentgateway
Best for: teams that want one gateway for LLM, MCP, and agent-to-agent connectivity.
agentgateway is an Apache-2.0 LF project with weighted, conditional, and priority-based routing across model providers. It supports API keys, JWT/OIDC, token and dollar budgets, health-aware failover, metrics, OTLP traces, and structured access logs.
A useful operational detail is that budget and token-limit behavior depends on when usage becomes known, so teams should test overshoot semantics and local-versus-global counters under real streaming workloads.
See the agentgateway documentation and model-failover guide.
6. Apache APISIX
Best for: organizations that already run APISIX or want AI traffic inside a mature API-gateway platform.
APISIX’s AI Gateway includes provider proxying, multi-instance routing, retries, fallback, token-aware rate limiting, prompt controls, AI observability, and Redis-backed exact or semantic caching.
APISIX 3.19.0 is the current release as of September 28, 2026. The project is Apache 2.0 licensed under the Apache Software Foundation.
Trade-off: it is broader than an LLM-only gateway. That is valuable for platform consolidation, but can be heavier than a dedicated proxy for smaller AI-native teams.
See the Apache APISIX AI Gateway.
7. Higress
Best for: Kubernetes teams that want API, model, and MCP traffic behind one Envoy/Istio-based gateway.
Higress is Apache 2.0 licensed and joined the CNCF Sandbox in March 2026. Its AI features include model routing, fallback, consumer authentication, token rate limiting, quotas, AI statistics, caching, dashboards, and MCP support.
It is especially attractive when an enterprise wants one gateway layer for conventional APIs, LLM traffic, and tool traffic rather than separate control planes.
See the Higress AI Gateway and AI token-rate-limiting documentation.
Routing, Authentication, and Observability: What Matters Most
For model-centric portability, LiteLLM, Portkey, and Bifrost provide the most direct abstractions around provider routing and fallback.
For cloud-native infrastructure, Agent Router and agentgateway bring routing, provider health, credentials, and quotas closer to Kubernetes and Gateway API operations.
For enterprises already standardized on an API gateway, APISIX and Higress can reduce architectural duplication by extending existing identity, traffic, logging, and deployment practices to AI workloads.
Authentication should be evaluated separately from provider-key storage. In production, test limits by user, application, team, model, RPM, TPM, token budget, and spend where supported. Also test what happens when one large streaming response crosses a limit after admission.
For logging, avoid storing full prompts and responses by default. Payload capture should follow data classification, privacy, redaction, retention, and incident-response policy.
Deployment and Licensing Considerations
The final shortlist has permissively licensed open-source cores:
- MIT: LiteLLM core, Portkey Gateway
- Apache 2.0: Bifrost, Agent Router, agentgateway, Apache APISIX, Higress
That does not mean feature parity with enterprise editions. Before approval, confirm whether SSO, advanced RBAC, audit logs, clustering, multi-region control, premium support, or administrative governance require a commercial tier.
Which Enterprise LLM Gateway Should You Choose?
Choose LiteLLM for broad provider coverage, model abstraction, virtual keys, budgets, and flexible routing.
Choose Portkey Gateway for lightweight, configuration-driven retries, fallbacks, routing, and guardrails.
Choose Bifrost for strong OSS virtual-key governance, semantic caching, Prometheus, and OpenTelemetry.
Choose Agent Router when Kubernetes plus Envoy is already your platform standard.
Choose agentgateway when model traffic is only one part of a broader MCP and agent-connectivity layer.
Choose APISIX or Higress when the goal is to converge API and AI gateway responsibilities.
Final Verdict
There is no universal best LLM gateway for every enterprise.
For many AI-native application teams, LiteLLM is the strongest general-purpose starting point because it combines provider breadth with practical virtual-key, budget, retry, routing, and fallback controls.
Bifrost is the strongest alternative when open-source governance and observability are central. Portkey is well suited to teams that value lightweight deployment and config-driven reliability.
For Kubernetes-first infrastructure, Agent Router and agentgateway deserve serious evaluation. If your enterprise already operates a shared API-gateway platform, APISIX and Higress may reduce duplication by extending the same control plane to AI traffic.
A good proof of concept should force provider failures, rotate keys, test streaming, exercise quota edges, validate telemetry, replay sensitive-data scenarios, and confirm that model swaps do not require application rewrites.
Build the Gateway Into the Right Enterprise AI Architecture
An LLM gateway solves the model-access layer, but production enterprise AI still needs application architecture, integration, observability, governance, deployment controls, and operational ownership.
LuMay’s AI Services support enterprise AI engineering, RAG, agents, integrations, deployment, and lifecycle management. Use the 8 Enterprise AI Architecture Patterns to place the gateway inside the wider stack, or book an architecture discussion with LuMay to scope a gateway proof of concept around your providers, identity model, observability stack, and deployment environment.
