To reduce LLM hallucinations in an enterprise application, ground answers in approved sources, let the system decline when evidence is weak, check every claim against its citation, and route risky answers to a person. Then measure all of it continuously.
No technique removes hallucinations entirely. The goal is fewer unsupported answers, and a system that shows its evidence and says "I don't know" when it should. This guide gives a five-layer method, a worked example and the metrics to track.
Why LLMs Hallucinate in Business Apps
NIST's Generative AI Profile (NIST AI 600-1) calls this risk confabulation: confidently stated but false or unsupported content. In enterprise apps, it usually comes from a handful of causes:
• Missing evidence: the right document was never indexed or retrieved
• Wrong evidence: an outdated policy or a similar but different record was retrieved
• Out-of-scope questions: the user asks something the sources cannot answer
• Pressure to answer: prompts and evaluations reward a confident reply over an honest abstention
• Long or messy context: the answer is in the context, but buried or contradicted
Most of these are retrieval and design problems, not model problems. That is good news, because they can be engineered.
The Five-Layer Method
Layer 1: Grounded retrieval. Index only approved, current sources, with version and owner metadata. Use hybrid search and reranking so exact terms such as product codes are found. Our guide to hybrid search for RAG covers this step.
Layer 2: Answerability rules. Before generating, decide whether the evidence is enough. If no passage passes a relevance threshold, or sources conflict, the system declines and offers a next step.
Layer 3: Constrained generation. Instruct the model to answer only from the provided passages, cite each claim, and say what is missing. Use structured output so citations are machine-checkable.
Layer 4: Post-generation checks. Verify that each cited passage supports the sentence that cites it. Managed options include Amazon Bedrock contextual grounding checks, which score grounding and relevance against thresholds you set.
Layer 5: Human review and monitoring. Send low-confidence or high-impact answers to a reviewer, log every decision, and feed corrections back into the test set.
Answerability Rules You Can Adopt
Illustrative example rule set for a policy assistant:
• Decline when no retrieved passage passes the relevance threshold
• Decline when the top passages come from documents marked superseded
• Ask a clarifying question when the query matches two different policies
• Answer with caveat when the only source is older than the review cycle
• Escalate when the question involves money, legal terms or safety
Write the rules down, version them and test them like code. They are business decisions, so a process owner should approve them.
Worked Example: HR Policy Assistant
Illustrative example. An HR assistant answers employee questions from 600 policy documents.
• Before: reviewers find answers that blend the 2025 and 2026 leave policies
• Layer 1 fix: add effective-date metadata and filter out superseded versions
• Layer 2 fix: decline when no 2026 passage passes the threshold, and link the HR contact page
• Layer 3 fix: require one citation per sentence in structured output
• Layer 4 fix: run a grounding check and block answers below the agreed score
• Layer 5 fix: route questions about pay or termination to HR staff
The team compares the share of unsupported answers on the same 250-question test set before and after each change. Numbers here are illustrative; your baseline will differ.
Metrics to Track
| Metric | What it tells you | How to measure |
|---|---|---|
| Unsupported claim rate | Share of answers with a claim not backed by its citation | Reviewer or automated check on a fixed test set |
| Retrieval hit rate | Whether the right passage was retrieved at all | Labeled question-to-document pairs |
| Correct abstention rate | Whether the system declines when it should | Test questions with no valid answer |
| Wrong abstention rate | Whether it declines answerable questions | Answerable test questions |
| Citation accuracy | Whether citations point to the supporting passage | Sentence-level review |
| Review turnaround | Whether escalations are handled in time | Ticket timestamps |
Track abstention both ways. A system that declines everything has no hallucinations and no value. For evaluation methods, see how to evaluate a RAG system and LLM evaluation metrics before launch.
When Fine-Tuning Helps, and When It Does Not
Fine-tuning can teach tone, format and when to abstain. It is a weak way to add facts that change, because the model cannot cite or update them. For changing knowledge, retrieval with citations is usually the better first step.
Who Owns Each Layer
Hallucination control fails when nobody owns a layer. A simple split:
• Content owners: keep sources current, retire superseded documents, approve answerability rules
• Engineering: retrieval quality, structured output, grounding checks and logging
• Risk or compliance: decide which topics always escalate and review incident trends
• Support or operations: handle escalations within the agreed response time
Review the metrics together monthly. A rising unsupported claim rate usually points to a content or retrieval change, so start the investigation there.
Questions to Ask Any RAG Partner
• How will you measure unsupported answers before and after launch?
• What happens when the sources do not contain an answer?
• How are citations checked, not just displayed?
• Who updates the test set when documents change?
Release Checklist
• Source list approved, versioned and owned
• Superseded documents excluded or labeled
• Answerability rules written and approved
• Citation per claim enforced in structured output
• Grounding check thresholds set and tested
• Test set includes unanswerable and trick questions
• Escalation path and reviewer response time agreed
• Monitoring dashboard tracks the six metrics above
Common Mistakes to Avoid
• Blaming the model first: most errors trace back to missing or stale sources
• Counting only accuracy: track abstentions, or the system learns to guess
• Citations nobody checks: a citation that does not support the claim is worse than none
• One-time testing: new documents and model updates change behavior
• Hiding uncertainty from users: show sources and caveats so people can verify
Limitations
Grounding checks estimate support; they do not prove truth, and they can be wrong.
Some vendor features, such as Azure's groundedness detection preview, are scheduled for retirement, so confirm status before you build on them. Every threshold in this guide must be tuned on your own data.
Next Step
If your assistant or agent answers from internal documents, LuMay's RAG development team can help design retrieval, answerability rules and the evaluation set that keeps hallucinations measurable.
• Amazon Bedrock contextual grounding check
