A RAG system can fail in two places. The retriever can fetch the wrong passages, or the model can misuse the right ones. A single "answer quality" score hides which one broke.
This guide shows how to evaluate a RAG system component by component. It covers the retrieval and generation metrics specific to RAG, how to build a labeled test set, tests that isolate failures, a sample scorecard and a pre-launch checklist.
It deliberately leaves out general LLM release metrics such as latency, cost and safety rates. Those are covered in our guide to LLM evaluation metrics before launch.
Why RAG Needs Its Own Evaluation
In a plain chatbot, a wrong answer has one suspect: the model. In RAG, there are at least four: the chunking, the retriever, the reranker and the generator. Each fix is different.
The RAG evaluation survey on arXiv frames evaluation the same way, splitting targets into retrieval and generation. Evidently's RAG evaluation guide also separates the two.
Step 1: Build a Labeled Test Set
Every metric below depends on a test set where you know the right passages and, ideally, the right answer.
• Question: written the way real users ask it, including typos and vague wording
• Relevant passage IDs: the chunk or document IDs that contain the answer
• Reference answer: a short, human-approved answer, where one exists
• Question type: single-fact, multi-passage, comparison, no-answer, permission-restricted
• Source date: so you can test freshness when documents change
Include questions the corpus cannot answer. A RAG system that invents answers when nothing relevant exists is a serious failure, and you only see it if you test for it.
Step 2: Score Retrieval
Retrieval metrics compare the passages returned for each question with the labeled relevant passages.
| Metric | Question it answers | When to watch it |
|---|---|---|
| Hit rate @k | Did at least one relevant passage appear in the top k? | Basic health check for every release |
| Recall @k | What share of all relevant passages appeared in the top k? | Multi-passage questions |
| Precision @k | What share of the top k passages were relevant? | When context windows fill with noise |
| MRR | How high was the first relevant passage, on average? | When only the top few passages reach the model |
| NDCG @k | Were the most relevant passages ranked highest? | After adding a reranker |
Pick k to match what you actually send to the model. If you pass five passages, Recall @20 tells you little about what the model sees.
Step 3: Score Generation Given the Context
Generation metrics ask whether the model used the retrieved passages well. The RAGAS paper defines faithfulness, answer relevance and context relevance as core reference-free measures.
• Faithfulness: share of claims in the answer supported by the retrieved context
• Answer relevance: does the answer address the question asked?
• Context utilization: did the model use the relevant passage it was given, or ignore it?
• Citation accuracy: does each cited source actually support the sentence it is attached to?
• Correctness vs reference: does the answer match the human reference answer?
Faithfulness and correctness are different. If the source document is outdated, an answer can be fully faithful and still wrong. Track both.
Step 4: Run RAG-Specific Failure Tests
These tests target behaviors that general LLM evaluation does not cover.
No-Answer Test
Ask questions the corpus cannot answer. The system should say it does not have the information. Score the share of correct abstentions. The survey above calls this negative rejection.
Conflicting-Source Test
Add two versions of a policy with different effective dates. Ask about the rule. A good system uses the current version or flags the conflict; a weak one blends both.
Permission Test
Ask, as a low-privilege user, a question answered only in a restricted document. The restricted passage must not be retrieved, quoted or summarized. Any leak is a release blocker.
Distractor Test
Insert passages that share keywords with the question but do not answer it. Check whether precision drops and whether the model is misled.
Component Swap Test
Feed the generator the labeled relevant passages directly, bypassing retrieval. If answers improve sharply, your problem is retrieval. If they stay poor, look at the prompt or model.
The component swap test is the fastest way to stop arguing about where the problem lives.
Sample RAG Evaluation Scorecard
The scorecard below is a template. The thresholds are illustrative placeholders, not recommended values; set your own from the risk of the use case. The "Example result" column shows how a team might fill it in.
| Area | Metric | How measured | Threshold (set your own) | Example result | Status |
|---|---|---|---|---|---|
| Retrieval | Hit rate @5 | Labeled passage IDs in top 5 | e.g., ≥ 0.90 | 0.86 | Fail |
| Retrieval | MRR | Rank of first relevant passage | e.g., ≥ 0.70 | 0.74 | Pass |
| Retrieval | Precision @5 | Relevant share of top 5 | e.g., ≥ 0.50 | 0.41 | Watch |
| Generation | Faithfulness | Claims supported by context, judge + human sample | e.g., ≥ 0.95 | 0.96 | Pass |
| Generation | Citation accuracy | Human check of cited passages | e.g., ≥ 0.95 | 0.89 | Fail |
| Generation | Correctness | Match to reference answer | e.g., ≥ 0.85 | 0.83 | Watch |
| Robustness | No-answer abstention | Unanswerable questions correctly declined | e.g., ≥ 0.90 | 0.78 | Fail |
| Robustness | Conflicting sources | Current version used or conflict flagged | e.g., 100% | 9 of 10 | Fail |
| Access | Permission leakage | Restricted content in any answer | 0 cases | 0 | Pass |
How to read this example: retrieval misses some answers (hit rate) and fills context with noise (precision). The generator is faithful but cites loosely. The system also guesses when it should abstain. Fix retrieval first, then citation formatting, then the abstention instruction.
Diagnosing From the Scorecard
• Low hit rate, good faithfulness: retrieval problem. Revisit chunking, hybrid search or the embedding model.
• Good hit rate, low faithfulness: generation problem. Tighten the prompt to answer only from context.
• Good hit rate, low context utilization: passages may be too long or ranked poorly; try a reranker or smaller chunks.
• Low abstention: add explicit "say you do not know" instructions and test again.
• Any permission leak: stop. Fix access filters at retrieval time, not in the prompt.
Our guides on combining keyword and vector search and RAG pipeline stages cover the retrieval-side fixes.
Using an LLM as a Judge
Faithfulness and relevance are often scored by another model, as RAGAS does. That scales, but a judge can be wrong.
○ Score a sample by hand and compare with the judge before trusting it
○ Keep the judge model and prompt versioned, like the system under test
○ Re-check agreement whenever you change the judge
○ Use humans for citation accuracy and for high-risk answers
Anyscale's RAG evaluation docs describe faithfulness as claims supported divided by total claims, which is easy for reviewers to audit.
When to Re-Run the Evaluation
• After changing chunking, embeddings, the index or the reranker
• After changing the prompt or model
• After a large batch of new or updated documents
• On a schedule, with fresh questions sampled from real usage
Keep the old test set as a regression suite and add new failures to it. A shrinking set of known failures is the clearest sign of progress.
Pre-Launch RAG Evaluation Checklist
• Labeled test set with relevant passage IDs and reference answers
• No-answer, conflicting-source, permission and distractor questions included
• Retrieval metrics reported at the k you actually send to the model
• Faithfulness and citation accuracy scored, with a human-checked sample
• Component swap test run to confirm where failures come from
• Thresholds set per metric and agreed with the business owner
• Zero permission leaks across all test users
• Judge model agreement with humans checked
• Scorecard and test set versioned with the release
Our Own Product Note
Disclosure: this is our own service. LuMay's AI development services include building and evaluating RAG flows. For tools that automate scoring, see our comparison of LLM evaluation tools.
Next Step
Copy the scorecard, label 50 of your real questions with their source passages and run the component swap test first. It will tell you where to spend the next sprint.
References
• Yu et al., Evaluation of Retrieval-Augmented Generation: A Survey (arXiv): https://arxiv.org/abs/2405.07437
• Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (arXiv): https://arxiv.org/abs/2309.15217
• Evidently AI, RAG evaluation guide: https://www.evidentlyai.com/llm-guide/rag-evaluation
• Anyscale, RAG evaluation: https://docs.anyscale.com/rag/evaluation
