TL;DR: Supervising AI legal review means testing accuracy before use, sampling output by risk, logging every decision and keeping a named attorney accountable. Ethics guidance treats AI output like work from a junior assistant: useful, but never final until checked.
Disclaimer: This article is general information, not legal advice. Professional duties vary by jurisdiction; consult your own regulator's rules and ethics counsel.
Supervising AI legal review is the set of controls a firm uses to make sure AI-generated findings, summaries or redlines are accurate, confidential and approved by a responsible lawyer before anyone relies on them.
Quick Answer: Validate the tool on matters with known answers, set review depth by risk, require source-linked output, log approvals, and keep a named attorney accountable for every deliverable. Revisit the controls whenever the tool, model or matter type changes.
Why Supervision Is an Ethics Issue, Not Just a Quality Issue
ABA Formal Opinion 512 says rules on competence, informed consent, confidentiality and fees principally apply to generative AI. Competence includes understanding the benefits and risks of the technology lawyers use.
In England and Wales, the SRA's compliance tips expect, as a minimum, the COLP to be responsible for regulatory compliance when new technology is introduced. It also highlights board oversight of purchasing and ongoing use.
Neither source bans AI. Both place responsibility on the lawyer and the firm, which is why supervision must be designed, not improvised.
Step 1: Validate Accuracy Before Live Use
- Build a test set
- Use closed matters where the correct answer is already known
- Include hard cases: poor scans, unusual clause wording, amendments
- Measure two error types
- Misses: issues the AI failed to flag
- False flags: issues it raised that are not real
- Misses are usually more dangerous in legal review
- Set a pass bar
- Agree in writing what error level is acceptable for each matter type
Frameworks such as the NIST AI Risk Management Framework, which is voluntary, encourage building trustworthiness into how AI is used and evaluated. A written test protocol is a practical way to start.
Step 2: Set Review Depth by Risk
| Approach | How it works | When to use | Risk |
|---|---|---|---|
| Full review | A lawyer checks every AI finding and every document | High-stakes, novel or litigated matters; new tools | Slowest, but lowest risk of missed issues |
| Risk-based sampling | Lawyer checks all flagged items plus a sample of clean ones | Routine review after validation | Misses in unsampled clean items |
| Exception-only review | Lawyer checks only items the AI flags | Low-risk, well-tested workflows | Highest risk if the AI misses issues |
Most firms start with full review, then move to sampling only after test results support it. Document why each matter type sits at its level.
Step 3: Require Verifiable Output
- Source links
- Every finding should point to the exact clause or page
- Playbook rules
- Codify your positions so the AI checks against firm standards; see our contract playbook template
- No silent edits
- AI redlines should be tracked changes, never accepted automatically
LuMay's Lexintis Legal Agents follow this pattern: outputs are linked to verified sources, and attorneys and billing partners retain full approval authority over generated actions.
Step 4: Protect Confidentiality
- Vendor terms
- Confirm data retention, model training use and sub-processors
- Client terms
- Check engagement letters and outside counsel guidelines for AI restrictions
- Access
- Limit tool access by matter and role, as with any document system
Step 5: Log, Sign Off and Bill Fairly
- Audit trail
- Record who reviewed which output, what changed and who approved it
- Named sign-off
- One responsible attorney approves each deliverable
- Fees
- Formal Opinion 512 notes lawyers generally cannot bill clients for learning a GAI tool
Billing controls can be automated too. BillingGuard checks rates, approvals, matter setup and billing requirements before invoices are submitted.
Common Supervision Failures
- Trusting a vendor demo instead of testing on your own matters
- Reviewing flagged items only, before validation supports it
- No owner for prompt or playbook changes
- Treating AI summaries as final in deposition or document review work
Key Takeaways
- Ethics guidance keeps responsibility with the lawyer and the firm
- Validate on known answers before live use; track misses and false flags
- Match review depth to risk, and document the reason
- Require source-linked output, audit trails and named sign-off; see first-pass risk flagging tools
Next Step
Want legal AI with citation-backed output and attorney approval built in? Book a Lexintis demo.
