TL;DR: Choose an open-source LLM evaluation framework around the failure you need to catch. DeepEval fits Python regression tests; Ragas fits retrieval-grounded evaluation; Promptfoo fits configuration-driven comparisons; Inspect AI fits multi-step tool tasks; LM Evaluation Harness fits model benchmarks; OpenAI Evals fits custom registry-based tests; and Evidently fits dataset-level reporting. There is no universal winner. Test two candidates on the same representative cases before standardizing.
An evaluation framework turns examples, model outputs and scoring rules into repeatable checks. The useful question is whether it can detect a failure that matters in your workflow and preserve enough evidence to investigate it.
This guide focuses on open-source code frameworks and libraries, rather than choosing an entire hosted evaluation platform. For broader collaboration and platform choices, see our enterprise LLM evaluation tools comparison.
Method and disclosure: This is an unranked shortlist based on official repositories and documentation checked on 10 November 2026. We reviewed documented evaluation workflows, licenses, deployment dependencies and practical fit; we did not run comparative benchmarks. LuMay publishes this guide and offers AI engineering services. The numbered order is for navigation.
Seven open-source frameworks at a glance
| Framework | Useful starting point | Core license | Key capabilities |
|---|---|---|---|
| DeepEval | Python application regression tests | Apache-2.0 | Test cases, assertion-style checks, application and agent metrics |
| Ragas | Retrieval-grounded application evaluation | Apache-2.0 | RAG evaluation workflows, configurable metrics, test generation |
| Promptfoo | Configuration-driven prompt and model comparisons | MIT | CLI evaluations, assertions, provider comparisons, CI/CD, red teaming |
| Inspect AI | Multi-step, tool-using evaluation tasks | MIT | Datasets, solvers, scorers, tool use, multi-turn tasks and logs |
| EleutherAI LM Evaluation Harness | Repeatable language-model benchmark tasks | MIT | Benchmark task configurations and multiple model backends |
| OpenAI Evals | Custom evaluations using a registry-based approach | MIT | Evaluation registry, templates, custom evaluation logic and completion functions |
| Evidently | Dataset-level evaluation reports and quality trends | Apache-2.0 | Python evaluation reports, tests, monitoring and self-hosted dashboard options |
Licenses above refer to the reviewed core repositories. Dataset, model-weight, dependency, hosted-service and enterprise-feature terms can differ. Open-source software can still incur model API, compute, storage and maintenance charges.
Key capabilities to check before choosing
- Application access: Can the runner call your actual application or a faithful test adapter, including its retrieval and tools?
- Scoring control: Can you combine deterministic assertions with model grading and human review?
- Reproducibility: Can you record the dataset, model, prompt, scorer and application versions?
- Failure evidence: Can engineers inspect individual cases and relevant execution steps?
- Automation: Can the runner produce results your release workflow can act on?
- Data boundaries: Where do inputs, traces, model calls, grading calls and reports go?
- Operating cost: Who owns runner maintenance, grader calibration and test-data refresh?
1. DeepEval: Python application regression tests
DeepEval is a practical starting point when your engineering team wants LLM checks to sit beside its existing automated tests. Its documentation describes pytest and CI workflows as well as metrics for task completion and tool correctness.
Key capabilities: Test cases, assertion-style checks, application and agent metrics.
Practical test: Create a test case for a support agent that must use the account-lookup tool before proposing an account change. Check the final answer and the tool behavior separately.
Limitations: A passing model-graded score is not proof that a business-system update was correct. Verify persisted state with deterministic checks. Judge calls can add cost and variability; shared reporting through Confident AI is a separate deployment decision.
Official source: DeepEval documentation and source.
2. Ragas: Retrieval-grounded application evaluation
Ragas is useful when the question is whether retrieved evidence supports an application's answers. Its repository provides RAG evaluation project templates and a metric-oriented workflow for iterative improvement.
Key capabilities: RAG evaluation workflows, configurable metrics, test generation.
Practical test: Keep a fixed set of policy questions and compare two retrieval configurations. Inspect the supporting passages and failed examples, rather than accepting one blended score.
Limitations: Generated test data can help expand coverage, but it should not replace real user questions or domain review. Verify which inputs each metric needs and which model or embedding endpoints it calls.
Official source: Ragas documentation and source.
3. Promptfoo: Configuration-driven prompt and model comparisons
Promptfoo suits teams that prefer declarative test configurations and command-line execution. Its repository documents automated evaluations, comparisons across providers, CI/CD checks and red-teaming workflows.
Key capabilities: CLI evaluations, assertions, provider comparisons, CI/CD, red teaming.
Practical test: Run a fixed set of customer-service inputs against a baseline and candidate prompt. Require valid structured output and check that restricted actions are refused.
Limitations: Local execution of the test runner does not mean all data stays local. A configured hosted model, grader or remote target can receive test content. Review every provider and reporting path before using confidential examples.
Official source: Promptfoo documentation and source.
4. Inspect AI: Multi-step, tool-using evaluation tasks
Inspect separates evaluation tasks into reusable components and supports tool use, multi-turn interaction and model-based grading. That structure is helpful when a single output string cannot describe the behavior being tested.
Key capabilities: Datasets, solvers, scorers, tool use, multi-turn tasks and logs.
Practical test: Evaluate an agent that must inspect a record, request approval and then perform a permitted action in a test environment. Score completion and prohibited behavior separately.
Limitations: A realistic tool environment and a trustworthy scorer require engineering work. Simulated tasks can miss production integration failures. Keep real credentials and irreversible actions outside experimental runs.
Official source: Inspect AI documentation and source.
5. EleutherAI LM Evaluation Harness: Repeatable language-model benchmark tasks
The LM Evaluation Harness is designed for evaluating language models on defined benchmark tasks. Its repository documents separate backends for Hugging Face models, vLLM and API-based models.
Key capabilities: Benchmark task configurations and multiple model backends.
Practical test: Compare candidate models on a relevant benchmark subset before spending time integrating every model into your enterprise application.
Limitations: A public benchmark result does not establish that a model can use your CRM safely or answer your policy questions correctly. Keep task settings, prompt formatting and model versions consistent, then run application-specific tests.
Official source: EleutherAI LM Evaluation Harness documentation and source.
6. OpenAI Evals: Custom evaluations using a registry-based approach
The open-source OpenAI Evals repository provides a framework and registry for evaluating LLMs and LLM systems. It documents existing evaluation templates, custom logic and a completion-function protocol for more advanced systems.
Key capabilities: Evaluation registry, templates, custom evaluation logic and completion functions.
Practical test: Use a small domain-specific dataset to make a previously observed failure reproducible, with an explicit expected outcome and scoring rule.
Limitations: This entry refers to the GitHub project. Check its current interfaces, model compatibility and execution dependencies before adoption. Do not assume a public benchmark or a repository default matches your application's release criteria.
Official source: OpenAI Evals documentation and source.
7. Evidently: Dataset-level evaluation reports and quality trends
Evidently provides an open-source Python framework for evaluating, testing and monitoring ML and LLM systems. It is useful when teams need to compare quality across datasets or track test results over time.
Key capabilities: Python evaluation reports, tests, monitoring and self-hosted dashboard options.
Practical test: Compare results for short questions, long conversations and different source-document groups. Investigate which segment regressed instead of relying on a global average.
Limitations: Its scope is broader than a dedicated agent test runner. Cloud and open-source features differ, so verify the reporting and collaboration workflow you need. Reports also require a reliable pipeline for collecting permitted evaluation data.
Official source: Evidently documentation and source.
Run a small bake-off before committing
Use the same application version, dataset and scoring rubric for both candidates. Include ordinary requests, ambiguous requests, missing evidence and actions that require approval. Start with a small reviewed dataset; expand it when you find failures. The exercise below is a proposed selection method, not a benchmark we performed.
| Check | Evidence to collect | Decision |
|---|---|---|
| Output validity | Schema errors and missing required fields | Use deterministic checks where possible |
| Task outcome | Expected record state or approved answer | Separate a convincing explanation from actual completion |
| Restricted behavior | Attempted unauthorized actions | Apply explicit blocking rules for critical failures |
| Judge agreement | Differences between human labels and automated grades | Revise the rubric before trusting the grade |
| Repeatability | Differences across repeated runs | Investigate instability rather than hiding it in an average |
| Runtime and cost | Runner duration and model/grader usage | Set an affordable cadence for routine checks |
A useful test artifact contains a case ID, permitted input, expected behavior, actual output, tool events, deterministic assertions, judge configuration, review notes and version identifiers. Avoid storing raw sensitive content when a permitted synthetic case can reproduce the problem.
Use RAG-specific evaluation methods when retrieval is central. For a broader release scorecard, use the LLM evaluation metrics guide.
Questions to resolve before adoption
Does open source mean air-gapped? No. Network isolation depends on the entire execution path, including model serving, graders, embeddings, telemetry and artifact storage.
Can one framework cover every test? Sometimes, but do not assume it. A benchmark harness, an application test runner and a reporting library serve different jobs. Use the smallest combination that covers your requirements.
Do automated grades replace subject-matter review? They can make review more targeted. Calibrate them against approved human labels, retain disagreements and revisit the rubric when the task changes.
Build an evaluation workflow around your application
Before selecting a framework, define the outcomes you must prove and the failures that must stop a release. Then choose the runner that exposes those failures with the least unnecessary infrastructure.
LuMay's AI engineering services can help scope application integrations, evaluation workflows and production operations. Bring a representative workflow, its access boundaries and a few known failures to the discussion.
