October 14, 2026 · LuMay AI
Live LLM applications fail in quiet ways. A chatbot can respond quickly and still give the wrong answer. An agent can finish a workflow while calling the wrong tool. A slow retrieval step may never raise an error.
This guide compares seven tools on tracing, evaluation, governance, deployment, integrations, and cost.
Monitoring, Tracing, and Evaluation: What's the Difference?
- Monitoring watches overall health: request volume, errors, latency, token use, and spend.
- Tracing records each step of one request: prompts, retrieval, model calls, and tool calls. A trace is the full record; a span is one step.
- Evaluation judges whether the answer or action was good: accurate, grounded, safe, and complete.
Tracing shows what happened. Evaluation shows whether it was acceptable. Automated judges help at scale, but validate their scores against human review before relying on them for release decisions.
How to read this guide: "Confirm" means the public documentation we reviewed did not settle the point. A missing mention does not prove a missing feature.
Quick Shortlist by Enterprise Requirement
- Open-source, self-hosted control: Langfuse, Arize Phoenix, MLflow, or Helicone.
- LangChain or LangGraph agents: LangSmith.
- OpenTelemetry-first tracing: Arize Phoenix or MLflow.
- Already standardized on Datadog: Datadog Agent Observability keeps AI traces beside infrastructure data.
- Gateway-led routing and cost visibility: Helicone.
- Existing ML platform on MLflow or Databricks: MLflow.
- Agent-failure analysis on CoreWeave: CoreWeave Agent Lens, currently in preview.
These are starting points by workload, not claims that the other tools lack these capabilities.
Technical Capabilities
| Tool | RAG and tool-call tracing | Evaluation | Tokens and cost | OpenTelemetry |
|---|---|---|---|---|
| Langfuse | Nested traces with tool and retrieval steps | Scores, LLM judge, datasets, experiments | Tokens and cost tracked | OTLP over HTTP; no gRPC yet |
| LangSmith | Runs, threads, agent steps | Online evals, annotation queues, Engine issue detection | Tracked | Confirm for your stack |
| Arize Phoenix | Retrieval, tool calls, sessions | Judges, datasets, annotations | Tokens and latency; dollar cost: confirm | Accepts OTLP |
| Helicone | Gateway requests and sessions; deep agent spans: confirm | Confirm | Cost and usage via gateway | Confirm |
| MLflow | Agent, retriever, and tool spans | Scorers, datasets, human feedback | Confirm | Fully compatible, ingest and export |
| Datadog | Spans for every chain and agent step | Quality, safety, prompt-injection checks; Patterns clustering | Cost, latency, and usage dashboards | GenAI semantic conventions |
| CoreWeave Agent Lens | Sessions, turns, LLM and tool calls, errors | LLM-as-a-judge metrics listed | Confirm | Accepts OTLP spans |
Deployment, Governance, and Cost
| Tool | Deployment | SSO and RBAC | Privacy and retention | Pricing basis |
|---|---|---|---|---|
| Langfuse | Open source self-hosted; Cloud in US, EU, JP, or HIPAA region | Teams add-on on Pro; included in Enterprise | Masking; retention management on Pro and above | Free Hobby; Core $29, Pro $199, Enterprise $2,499 per month, plus usage units |
| LangSmith | Cloud (US or EU); hybrid and self-hosted on Enterprise | Google/GitHub SSO on Plus; custom SSO and RBAC on Enterprise | Base traces 14 days; extended 180 days | Free Developer seat; Plus $39 per seat per month, plus traces |
| Arize Phoenix | Free self-hosting under Elastic License 2.0 | Confirm; Arize AX is a separate commercial product | Confirm | No license fee; you pay for hosting |
| Helicone | Hosted gateway; open-source code for self-hosting | Confirm | Confirm | Plans plus request and storage usage; provider costs at 0% markup |
| MLflow | Open source self-hosted; managed MLflow via Databricks | Depends on your hosting setup | PII redaction and sampling built in | Free open source; hosting and managed-platform costs |
| Datadog | SaaS only; not on two US government sites | Datadog platform controls | Automatic sensitive-data scanning and redaction | Metered by ingested LLM spans |
| CoreWeave Agent Lens | Hosted preview | Confirm | Confirm | Listed with W&B Weave plans; ingestion billed per MB |
Tool Profiles
1. Langfuse: Open-Source LLM Engineering Platform
- Best fit: Teams wanting LLM-specific tracing and evaluation with a choice of self-hosted or managed cloud.
- Strengths: Langfuse logs prompts, responses, tokens, latency, and tool or retrieval steps as nested traces. It adds cost tracking, dashboards, alerts, scores, LLM-as-a-judge, datasets, and prompt management.
- Integrations: Python and JavaScript SDKs, OpenAI, LangChain, LlamaIndex, and OpenTelemetry for other languages.
- Deployment: Open source and self-hostable, or Langfuse Cloud in US, EU, Japan, or HIPAA regions.
- Limitations: OpenTelemetry ingestion is HTTP only, with gRPC not yet supported. Enterprise SSO and fine-grained RBAC need the Teams add-on ($300 per month) or Enterprise. Self-hosting adds operational work.
- Pricing basis: Free Hobby tier; Core from $29, Pro from $199, and Enterprise from $2,499 per month, plus usage units beyond the included amount.
2. LangSmith: Tracing Tied to Evaluation Workflows
- Best fit: Teams building on LangChain or LangGraph, or wanting one platform from development to production.
- Strengths: LangSmith offers dashboards, alerts, online evaluations, annotation queues, and Engine, which detects recurring issues and diagnoses root causes.
- Integrations: OpenAI, Anthropic, CrewAI, Vercel AI SDK, Pydantic AI, and others.
- Deployment: Cloud in US or EU. Hybrid and self-hosted options are Enterprise only.
- Limitations: Online evaluators upgrade scored traces to extended retention by default, which raises trace charges. Configure this deliberately.
- Pricing basis: Developer is free for one seat; Plus is $39 per seat per month with 10,000 base traces included, then pay as you go. Enterprise is custom.
3. Arize Phoenix: Open Tracing for RAG Debugging
- Best fit: Engineers diagnosing retrieval failures, embeddings, tool arguments, and output quality.
- Strengths: Phoenix accepts traces over OTLP and shows retrieved documents, tool calls, sessions, token usage, latency, and annotations.
- Integrations: OpenTelemetry-based instrumentation for LangChain, LlamaIndex, DSPy, and major model SDKs.
- Deployment: Self-hosting is free with no feature gates, under the Elastic License 2.0. Review the license terms with your legal team.
- Limitations: Phoenix is separate from Arize's commercial AX platform. Do not assume AX enterprise governance is included. Dollar-cost tracking and SSO need confirmation.
- Pricing basis: No license fee for self-hosting; budget compute, storage, and maintenance.
4. Helicone: Gateway-Centric Usage and Cost Visibility
- Best fit: Teams routing many model providers through one gateway.
- Strengths: The Helicone quick start describes an OpenAI-compatible gateway with access to 100+ models, automatic request logging, provider fallbacks, unified billing, and bring-your-own provider keys.
- Integrations: Standard OpenAI SDK clients pointed at its endpoint, across providers such as OpenAI, Anthropic, Vertex, and Groq.
- Deployment: Hosted gateway; the code is open source for self-hosting.
- Limitations: A gateway sees traffic passing through it, so internal retrieval and tool steps may need extra instrumentation. Evaluation and OpenTelemetry support need confirmation. A gateway is also a critical control point, so review its security and failure behavior carefully.
- Pricing basis: Credits for model usage at 0% markup on provider charges, plus plan fees tied to request volume and storage. Confirm current rates with Helicone.
5. MLflow: Observability Within the ML Lifecycle
- Best fit: Teams already using MLflow or Databricks for experiments and model management.
- Strengths: MLflow Tracing is fully OpenTelemetry-compatible for ingest and export, with PII redaction, sampling, evaluation, and human feedback alongside.
- Integrations: OpenAI, Anthropic, LangChain, LlamaIndex, DSPy, Pydantic AI, and Vercel AI.
- Deployment: Self-hosted on your own infrastructure, or managed MLflow through Databricks.
- Limitations: You run the server. SSO, access control, and retention depend on how you host it. Token and cost views need confirmation.
- Pricing basis: Free open source; budget hosting, operations, and any managed-platform charges.
6. Datadog Agent Observability: AI Traces With Operations Context
- Best fit: Enterprises already running Datadog for infrastructure and application monitoring.
- Strengths: Datadog's documentation lists step-by-step traces, cost and latency dashboards, Insights for recurring problems, automatic sensitive-data scanning and redaction, prompt-injection detection, and Patterns topic clustering.
- Integrations: Python SDK for OpenAI, LangChain, AWS Bedrock, and Anthropic, plus OpenTelemetry GenAI semantic conventions.
- Deployment: SaaS only. Not supported on two Datadog US government sites.
- Limitations: Billing is per ingested LLM span, and one agent workflow can create several spans. Model costs at your real volume.
- Pricing basis: Metered by ingested LLM spans. Check current rates on Datadog's pricing page.
The product was formerly called LLM Observability; Datadog's documentation now uses "Agent Observability."
7. CoreWeave Agent Lens: Agent Behavior and Failure Patterns
- Best fit: Teams running agents on CoreWeave or wanting agent-focused failure analysis.
- Strengths: Agent Lens traces conversations, turns, LLM calls, tool calls, reasoning, and errors.
- Integrations: Built-in agent framework integrations, or send OpenTelemetry spans to its OTLP endpoint with no Agent Lens SDK required.
- Deployment: Hosted service, labeled "Preview" on CoreWeave's pricing page.
- Limitations: Preview status. SSO, RBAC, retention, and export controls are not documented on the pages we reviewed. Confirm its relationship to W&B Weave and its roadmap before committing.
- Pricing basis: Listed alongside W&B Weave across Free, Pro, and Enterprise plans, with data ingestion billed per MB.
Assessed Trade-offs
These points are our assessment, not vendor claims.
- Self-hosting: Cuts license spend but adds databases, upgrades, scaling, and security patching. Self-hosting is not automatically more secure.
- Managed cloud: Speeds adoption, but your security team must review data residency, retention, and what content enters vendor storage.
- Gateways: Quick to adopt and strong on cost data. Deep retrieval and tool traces may need extra instrumentation.
- Platform-native tools: One vendor reduces tool sprawl, but evaluation depth may lag specialist products.
- Preview products: Treat them as pilot candidates until deployment and security terms are confirmed in writing.
What Drives Operating Cost
- Trace volume: Agents generate many spans per task, and most tools bill by traces, spans, or requests.
- Retention: Longer storage costs more and widens privacy exposure.
- Sampling: Keep every error trace and sample the rest.
- Judge calls: Automated evaluation consumes model tokens, so budget for it.
- People: Instrumentation, self-hosting, and security reviews take engineering time.
Selection Checklist
Ask each vendor to demonstrate:
- Coverage: Retrieval chunks, tool inputs and outputs, retries, failures, and agent handoffs in one connected trace.
- Quality: Custom judges, human annotation, regression tests, and judge calibration.
- Governance: SSO, least-privilege roles, PII masking, data region, deletion, and audit logs.
- Portability: OpenTelemetry in, trace and dataset export out.
- Economics: Spans per request, sampling, retention, evaluator tokens, overages, and staff time.
Run a Practical Four-Week Pilot
Use one RAG assistant and one tool-using agent, with production-like but approved or sanitized data. Test two finalists side by side, ideally through OpenTelemetry.
- Week 1: Instrument. Capture trace IDs, model and prompt versions, retrieved source IDs, tool status, latency, and tokens.
- Week 2: Inject failures. Send representative traffic plus known problems: stale documents, tool timeouts, invalid arguments, and prompt-injection attempts. Check that each is visible and traceable.
- Week 3: Evaluate. Score a sample of traces automatically, then have domain experts review a subset and compare agreement.
- Week 4: Verify controls and cost. Test SSO, roles, masking, retention, deletion, and export. Forecast spend at normal and peak volume.
Decision gate: Choose the tool that makes failures easiest to find and fix, within your security requirements and budget.
How LuMay Supports Enterprise LLM Observability
Tools do not fix reliability by themselves. The operating model around them matters as much: what to trace, how to evaluate, who responds, and how data stays protected.
- Engineering: LuMay's LLM development services document evaluation, monitoring, versioning, regression testing, and rollback procedures as part of LLMOps.
- Operations: Its managed AI services cover ongoing monitoring, optimization, and support.
- Governance: The security page lists SSO, role-based access, audit trails with end-to-end tracing, PII masking, and human approval gates.
- Delivery: The services hub spans integration, automation, RAG, and agents.
LuMay acts as an implementation partner, not a reseller or certified partner of these seven tools. Confirm tool-specific experience during discovery. Contact LuMay to plan a vendor-neutral observability pilot.
