How to Evaluate RAG Systems in Production: Metrics and Test Cases
A RAG demo can answer ten questions convincingly and still fail in production. The failure may come from missing documents, wrong access filters, stale chunks, a misleading answer, or a correct answer with an unusable citation. Evaluate these failures separately, rather than reporting one overall 'accuracy' number.
This guide is for engineering leads and teams operating retrieval-augmented generation (RAG) applications. It provides a reproducible evaluation plan, not a universal benchmark or a claim about LuMay customer results.
What should a production RAG evaluation measure?
Measure retrieval, answer quality, operational behavior, and security boundaries as separate layers. Microsoft's RAG evaluator documentation distinguishes document retrieval and answer-oriented evaluators; Amazon Bedrock's evaluation documentation also separates context relevance, coverage, faithfulness and citation measures.
| Layer | Question to answer | Useful evidence |
|---|---|---|
| Retrieval | Did the system find the necessary source material? | Recall@k, ranking quality, source identifiers |
| Generation | Is every material claim supported? | Human-checked groundedness and factual correctness |
| Citation | Can a user locate the supporting passage? | Citation precision and coverage |
| Operations | Does the service behave within its limits? | p50/p95 latency, cost per request, timeout and error rates |
| Access control | Can users only retrieve authorized content? | Denied cross-tenant and restricted-document test cases |
Step 1: Build a representative golden dataset
Collect real user task patterns with private data removed. Include simple lookup, multi-document synthesis, ambiguous questions, absent-answer questions, conflicting versions and restricted-content requests.
For each test case, store the query, user role, corpus snapshot, relevant source IDs, expected key facts, allowed response behavior and reviewer notes. Keep queries and ground truth versioned with the application release.
Illustrative case: A procurement user asks which contract amendment changed renewal notice terms. The expected evidence is a particular amendment and clause, not just a plausible answer about renewals. If the amendment is absent, the system should say it cannot verify the answer.
Step 2: Evaluate retrieval before rewriting prompts
Measure whether the relevant passage appears in the top results. When you have labeled relevant documents, recall@k and ranking-based metrics help pinpoint retrieval problems. When labels are unavailable, review context relevance with a human sample or an evaluator whose limitations you document.
Log retrieved document IDs, versions and scores. Test filters for tenant, department, sensitivity, language and effective date. A retrieved chunk that is highly similar but obsolete is not a successful result.
If retrieval fails, inspect ingestion, parsing, chunk boundaries, metadata, filters and reranking before tuning generation.
Step 3: Evaluate answers against the retrieved evidence
For each answer, ask:
- Is the requested question answered, including its qualifiers?
- Are assertions supported by the retrieved passages?
- Are important contradictions or missing facts disclosed?
- Do citations point to the passages that support the claims?
- Does the response abstain when authoritative evidence is absent?
An LLM judge can triage cases, but high-risk failures need human adjudication. Keep the rubric and judge version fixed when comparing releases.
Step 4: Add negative and adversarial cases
Test deleted and superseded records, malformed PDFs, tables that span pages, prompt injection hidden in retrieved text, and cross-tenant lookups. For sensitive systems, a good answer delivered from unauthorized context is still a failure.
Run these tests whenever documents, retrieval settings, models, prompts or permissions change.
Step 5: Set release gates and monitor drift
Define acceptance criteria from the application's business risk rather than copying another team's threshold. For example, a low-risk knowledge assistant may tolerate a recoverable citation issue; a legal or financial workflow may require reviewer sign-off for every material unsupported claim.
Track a stable regression set alongside a rotating sample of recent queries. Investigate changes in source coverage, unsupported-claim frequency, response time and cost. Review failed examples, not only aggregate scores.
A compact evaluation record
| Field | Example of what to record |
|---|---|
| Test ID | procurement-amendment-017 |
| Corpus version | document snapshot ID |
| Role and filters | procurement-reader; tenant A |
| Expected source | amendment identifier and clause |
| Retrieval outcome | relevant passage found or missing |
| Answer outcome | correct, partial, unsupported or abstained |
| Citation outcome | passage verified or not |
| Reviewer and date | reviewer ID; assessment timestamp |
The record above is an illustrative template, not a reported LuMay benchmark.
Common mistakes to avoid
Do not test only friendly questions, assume a fluent answer is correct, compare scores across changing evaluation rubrics, or treat LLM-judge outputs as verified ground truth. Do not optimize latency by disabling authorization checks or bypassing source validation.
Next step
Start with a small, reviewed test set representing the decisions users actually make. Expand it whenever a production failure reveals a new failure class. For adjacent implementation risks, see LuMay's document parsing guide and incremental indexing practices.
Frequently asked questions
Is retrieval recall enough to evaluate RAG?
No. Relevant documents can be retrieved while the generated response still misstates their meaning or cites the wrong passage.
Should every answer have a reference answer?
Not necessarily. Ground-truth references help reproducibility, but contextual relevance and evidence-supported claims can also be reviewed without one fixed answer.
How often should a RAG system be reevaluated?
Reevaluate after changes to the model, corpus, parser, retrieval logic, permissions or prompt, and sample production behavior on an ongoing schedule.
