To validate AI legal document review, measure how many relevant documents the system found (recall), how many of its picks were actually relevant (precision), and how many relevant documents remain in the discarded set (elusion), using a random sample reviewed by qualified people.
Validation is not optional. Lawyers remain responsible for the result, and ABA Formal Opinion 512 says lawyers using generative AI must understand its capabilities and limits and review its output appropriately.
This guide explains the metrics, a sampling method, a worked calculation and the documentation that makes a review defensible. It is general information, not legal advice.
The Three Metrics That Matter
| Metric | Question it answers | Formula |
|---|---|---|
| Recall | Of all relevant documents, how many did the review find? | Relevant found ÷ total relevant |
| Precision | Of documents marked relevant, how many truly are? | Truly relevant ÷ marked relevant |
| Elusion | Of documents set aside, how many are actually relevant? | Relevant in discard sample ÷ discard sample size |
Recall matters most when missing documents is costly, as in discovery production. Precision drives review cost, because low precision sends irrelevant documents to human reviewers.
Step 1: Define Relevance Before Measuring
Write a review protocol that defines relevance, privilege and any issue codes, with examples. Without a shared definition, reviewers will disagree and every metric becomes noise.
• Have the supervising attorney approve the protocol
• Train reviewers on the same examples
• Measure agreement between two reviewers on a small set and resolve conflicts
Step 2: Draw Random Samples
Validation relies on random samples, not documents chosen because they look interesting. Sample from both the set the AI marked relevant and the set it set aside.
ACEDS guidance on TAR validation explains that sample size depends on the confidence level and margin of error you want, and that prevalence of relevant documents affects how many samples you need.
• Use a documented random method and record the seed or tool used
• Keep reviewers blind to the AI's call where practical
• Have qualified people make the final relevance decision
Step 3: Calculate and Interpret
Combine sample results to estimate recall. A low elusion rate in the discard pile is strong evidence that few relevant documents were missed.
Report estimates with their confidence intervals, not just a single number. A recall estimate from a small sample can be far less certain than it looks.
Worked Example: Estimating Recall From Samples
Illustrative example: hypothetical figures to show the arithmetic, not a real matter or benchmark.
| Item | Value |
|---|---|
| Total documents | 100,000 |
| AI marked relevant | 20,000 |
| AI set aside | 80,000 |
| Sample of marked relevant reviewed | 400, of which 340 relevant |
| Sample of set-aside reviewed | 1,500, of which 15 relevant |
Precision estimate: 340 ÷ 400 = 85%. Estimated relevant in the marked set: 20,000 × 85% = 17,000.
Elusion estimate: 15 ÷ 1,500 = 1%. Estimated relevant left behind: 80,000 × 1% = 800.
Recall estimate: 17,000 ÷ (17,000 + 800) = about 95.5%. Before relying on it, calculate confidence intervals and have the supervising attorney decide whether the result is reasonable for the matter.
Step 4: Document for Defensibility
Courts generally look at whether a process was reasonable, not perfect. Federal Rule of Civil Procedure 26(b)(1) frames discovery in terms of relevance and proportionality, and Rule 26(g) requires a reasonable inquiry behind certifications.
• Record the protocol, tools, versions and settings
• Keep sample sizes, random method and results
• Log reviewer decisions and disagreements
• Note any changes mid-review and why
• Agree protocols with opposing counsel where the matter requires it
Courts have accepted technology-assisted review when the process is transparent; Da Silva Moore v. Publicis Groupe (S.D.N.Y. 2012) is a commonly cited early example.
Professional Duties to Keep in View
ABA Formal Opinion 512 connects generative AI use to existing Model Rules. State bars may add their own guidance, so check your jurisdiction.
• Rule 1.1, competence: understand how the tool works and where it fails
• Rule 1.6, confidentiality: assess how the tool stores and uses client data
• Rules 5.1 and 5.3, supervision: supervise lawyers and nonlawyers, including vendors, using the tool
• Rule 1.5, fees: bill reasonably when AI reduces time spent
Common Mistakes to Avoid
• Sampling only the AI's positives: without a discard sample you cannot estimate what was missed
• Changing the protocol mid-review without re-sampling: earlier estimates no longer apply
• Reporting a single number: recall without a confidence interval overstates certainty
• Letting the tool grade itself: final relevance calls need qualified human reviewers
• Treating privilege as relevance: privilege review needs its own checks and logs
Questions to Ask Any Review Vendor
• How do you recommend sampling, and can we export the sample and decisions?
• Which settings and model versions were used, and are they logged?
• Can we rerun validation on our own documents?
• How is client data stored, retained and excluded from training?
How We Evaluated This Guidance
We reviewed current vendor and standards documentation listed under Sources, read on October 31, 2026, and compared it with the pages ranking for this topic. We did not run benchmarks; figures marked Illustrative example are assumptions, not results.
Limitations
Metrics measure agreement with your reviewers' definition of relevance. If the protocol is unclear, high scores can hide real misses. Generative tools that summarize or extract also need checks for factual accuracy and invented content, not just relevance.
No vendor metric replaces validation on your own documents. Ask vendors how results were measured, on what data, and whether you can rerun the test yourself.
Our own product (disclosure: LuMay sells this). LuMay's Lexintis legal AI page describes privilege-aware document review with citations preserved for every recommendation.
Accuracy on your matters is Not verified until you validate it with a method like the one above. This is not a ranking.
Key Takeaways
• Define relevance in a written protocol before measuring anything
• Sample both the AI's relevant set and its discard set at random
• Report recall, precision and elusion with confidence intervals
• Document tools, settings, samples and decisions for defensibility
• Keep final calls with qualified reviewers under attorney supervision
Next Step
Build a validation protocol before your next AI-assisted review. Review Lexintis document review workflows with LuMay, and see our guide on how to verify AI-generated legal research citations.
Sources
• ABA Formal Opinion 512: Generative Artificial Intelligence Tools
• Federal Rules of Civil Procedure, Rule 26
• ABA Model Rules of Professional Conduct
• ACEDS: TAR validation and sample size
• Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012), citation only; link not verified
