8 Data Readiness Checks Before Building Enterprise Generative AI

Data readiness for generative AI means your information is owned, lawfully usable, accurate enough, current, and retrievable for the questions a defined use case must answer. It also means you can prove each point with evidence before engineering begins.

Data volume is not readiness. If documents are unowned, permissions are unclear, or the right answers were never written down, no model choice will fix it.

This checklist gives CIOs, CTOs, and data leaders eight tests to run first. Each names the risk, the test, the evidence, the accountable owner, and an illustrative pass criterion.

All examples are hypothetical, and all thresholds are illustrative. Agree real acceptance criteria with business, security, and data owners before testing.

For broader risk framing, NIST's Generative AI Profile (NIST AI 600-1) is a useful companion. It is a voluntary, cross-sector resource that applies the AI Risk Management Framework to generative AI.

First, decide what you are building: RAG, fine-tuning, or neither

Retrieval-augmented generation (RAG) retrieves approved content at question time and passes it to a model as context. It suits policy assistants, document Q&A, and knowledge search, where facts change and answers need citations.

Fine-tuning adapts how a model behaves, such as output format, terminology, or a narrow classification task. It needs suitable labeled examples plus separate evaluation data, and changed facts require another training run.

Many enterprise projects never train a model at all. A capable commercial or open-source model with well-prepared retrieval is often enough. LuMay's overview of LLM development options compares APIs, RAG, fine-tuning, and private deployment.

The checks below apply to both paths. Where fine-tuning adds work, we say so.

1. Source inventory, ownership, and lineage

If no one owns a source, no one fixes its errors, approves its use, or answers for a wrong response. Lineage, the record of where data came from and how it changed, lets you trace a disputed answer back to its origin.

Separate the system of record from copies made for search. A copy may be useful, but its existence does not make it authoritative.

• Test: Inventory every source the workflow needs, then trace sampled records from origin through extraction and indexing.
• Evidence: A signed-off source register, lineage notes, inclusion decisions, and documented authority rules.
• Owner: Domain data owners, coordinated by the chief data officer or data governance lead.
• Pass (illustrative): Every in-scope source has a named owner, a traceable origin, and an agreed place in the authority hierarchy.

Hypothetical example: A procurement assistant retrieves a regional handbook that conflicts with corporate policy. Both are authentic, but nobody has defined which one governs the transaction.

Fix: Assign owners, resolve authority, and quarantine unexplained copies. If sources sit in systems without modern APIs, plan connectors to existing enterprise systems early.

For fine-tuning, lineage matters even more. Once examples shape model weights, removing one is hard, so record exactly what entered each training set.

2. Usage rights, privacy, and access permissions

This is a blocker check. Access to a source does not establish approval to process, index, retain, or train on it.

Microsoft's RAG preparation guidance advises identifying security constraints early and retaining per-user access permissions in your chunking solution. Fine-tuning raises the bar again, because rights must cover training, not only search and display.

• Test: Review licenses, contracts, and privacy terms; then query as different roles, external users, and recently revoked accounts.
• Evidence: Legal and privacy sign-off, data classification, a role-based access matrix, and failed-access test results.
• Owner: The privacy or data protection lead, with legal and information security sign-off.
• Pass (illustrative): Rights are resolved, and restricted content never reaches retrieval results, citations, caches, or logs.

Inspect retrieved passages, citations, and caches, not only the final answer. A response can refuse correctly while the interface still shows a restricted document's title and excerpt.

Hypothetical example: An HR assistant declines to state a colleague's salary. Its source panel still displays the salary spreadsheet's name and an excerpt to an unauthorized employee.

Fix: Enforce authorization before content reaches the model, minimize sensitive fields, and retest revocation. A prompt telling the model to keep secrets is not access control.

3. Accuracy, completeness, duplicates, and contradictions

A model grounded in wrong or conflicting content will repeat it fluently. Missing units, repeated documents, and contradictory instructions often matter more than many cosmetic errors.

• Test: Profile required fields and duplicates, then sample records against authoritative originals with domain experts.
• Evidence: A defect register showing severity, affected sources, reconciliation rules, and before-and-after samples.
• Owner: Business data stewards, supported by data engineering.
• Pass (illustrative): Sampled errors below an agreed tolerance, such as 2%, with no unresolved critical contradictions.

Sample by risk, not convenience. A low overall error rate can hide a serious problem in the small subset used for high-impact decisions.

Hypothetical example: Two product sheets list different operating limits because one covers an older model. Deleting it as a duplicate would erase valid historical information.

Fix: Correct verified errors, preserve meaningful distinctions with applicability metadata, and route uncertain material to review. Cleaning reduces source errors, but it does not eliminate hallucinations.

4. Business-question coverage and representativeness

Clean data can still miss what users actually ask. Head-office documentation may overlook regional practices, specialist terminology, other languages, or exception handling.

The same Microsoft guidance recommends gathering test questions alongside representative content, including questions the documents cannot answer. Those unanswerable questions let you design the fallback.

• Test: Collect real user questions; map each to its supporting passage or mark it unanswerable.
• Evidence: A question-to-source coverage matrix, segmented by role, region, product, and language.
• Owner: The business product owner, with domain experts.
• Pass (illustrative): Every critical question has reliable evidence or an approved abstention or escalation path.

Hypothetical example: An HR assistant answers standard leave questions well. It cannot distinguish procedures for shift workers and office staff, so average scores hide a predictable failure.

Fix: Add authoritative content, narrow the scope, or design a clear "I don't know" response with human escalation.

If your inventory already shows gaps across several systems, share your use case and data constraints with LuMay's engineers to scope the remediation.

5. Freshness, versioning, updates, and deletions

Correct information becomes unreliable once it expires. Deleting or restricting a source is not enough if extracted copies, embeddings, or caches still serve it.

• Test: Edit, supersede, revoke, and delete test items at source; then inspect indexes, embeddings, caches, and citations.
• Evidence: Version identifiers, effective dates, ingestion timestamps, sync logs, and deletion test records.
• Owner: The data platform owner, with source owners setting acceptable staleness per content type.
• Pass (illustrative): Policy updates appear within an agreed window, such as four hours; revoked access is enforced immediately.

Set freshness per source. A reference manual and a live inventory balance should not share one refresh schedule, and some questions need a direct query to the operational system.

Hypothetical example: A support assistant cites last month's return policy. The new version was added, but the old chunks were never retired.

Fix: Use stable identifiers, supersession rules, and change data capture, which tracks inserts, updates, and deletes at source. Monitor sync jobs and test deletion paths explicitly.

If you fine-tune, removing retrieval records does not reverse training. Changed facts mean retraining, which is one reason current knowledge usually belongs in retrieval.

6. Extraction, OCR, tables, and metadata

A file people can read may produce unusable text. Optical character recognition (OCR) can misread scans, and extraction can detach table values from headings, units, or footnotes.

• Test: Sample every format and layout; compare extracted output with originals, especially tables, signs, and reading order.
• Evidence: Original-to-extracted comparisons, field-level accuracy, parser versions, and a human-review exception queue.
• Owner: The ingestion or document-processing engineering lead, validated by domain experts.
• Pass (illustrative): Critical fields extracted at an agreed accuracy, such as 98%; unsupported layouts are rejected or reviewed.

Check metadata as carefully as body text. Title, source, owner, version, effective date, language, region, and access level drive filtering, permission checks, and citations.

Hypothetical example: A scanned pricing table loses its "per quarter" heading. A correct number now reads as a monthly charge, in perfectly fluent text.

Fix: Use layout-aware parsing, repeat table headers across chunks, and attach source metadata to every chunk. Keep generated summaries distinguishable from verbatim source text.

7. Retrieval, chunking, embeddings, and citations

Good documents still fail if retrieval returns the wrong passage. Chunking splits content into retrievable sections; embeddings convert text into numeric vectors so the system can search by meaning.

Microsoft's chunking guidance explains that oversized chunks add noise and cost, while undersized ones lose context. Careless splitting can also separate a rule from its exception.

• Test: Run coverage questions through retrieval alone; check that complete supporting evidence appears in the top results.
• Evidence: Recall-at-k scores, relevance judgments, citation checks, configuration versions, and failure categories.
• Owner: The AI or RAG engineering lead, with domain experts judging relevance.
• Pass (illustrative): Complete evidence in the top five results for at least 85% of answerable questions; correct citations for every critical claim.

Recall at k measures how often the right passage appears among the top k results. Log retrieval misses separately from generation errors, because they need different fixes.

Hypothetical example: A maintenance instruction is retrieved without its preceding safety condition. The answer cites the right document but supports a procedure that should not apply.

Fix: Chunk on document structure, preserve headings and exceptions, and test hybrid keyword-plus-vector search or reranking. See how a RAG development pipeline handles permission-aware retrieval and citations.

Hybrid search matters most for product codes, clause numbers, and other exact terms that embeddings can blur.

8. Held-out evaluation, leakage prevention, security testing, and monitoring

Testing on the questions used to tune prompts or train models inflates scores; this is leakage. Production also brings attacks, including malicious instructions hidden inside retrieved documents.

NIST's profile and Microsoft's evaluation guidance both flag this indirect prompt injection risk. The OWASP Top 10 for LLM applications lists prompt injection and vector and embedding weaknesses among its 2025 risks.

• Test: Freeze a held-out question set nobody tunes against; run adversarial, permission, and unsupported-question tests with real user roles.
• Evidence: Groundedness, correctness, and completeness results, red-team findings, a monitoring plan, and a release decision.
• Owner: An AI assurance lead for evaluation, security for red-teaming, and operations for monitoring.
• Pass (illustrative): Held-out scores meet agreed target ranges, no critical security finding is open, and escalation and rollback are tested.

Model outputs vary between runs, so Microsoft recommends target ranges over single numbers. For RAG, source documents can stay searchable; only evaluation questions, answers, and scoring decisions stay out of tuning.

Hypothetical example: A copilot improves after engineers tune prompts against every benchmark question. It then fails unfamiliar questions, because the score measured adaptation to the benchmark.

Fix: Rebuild an independent evaluation set, isolate untrusted content from instructions, and rerun tests whenever models, prompts, or sources change. This operating discipline is core to AI engineering for production systems.

For fine-tuning, split training and evaluation data by document, entity, or time so overlapping examples cannot leak.

How to sequence the checks

Run checks 1 and 2 first, because their failures stop the project. There is little value in cleaning documents you may not be allowed to use.

Checks 3 to 6 can run in parallel on a representative sample. Checks 7 and 8 need a working retrieval prototype, so schedule them once it exists.

Record results in one shared register. A readiness review is far easier when each check has dated evidence and a named approver.

Data readiness checklist at a glance

Go/no-go blockers

A readiness average must not hide a failed mandatory gate. Stop deployment while any of these remain:

• Usage rights, processing approval, or required privacy decisions are unresolved.
• Unauthorized content reaches retrieval results, answers, citations, caches, or logs.
• No named owner exists for content the system will cite.
• Critical questions lack reliable evidence and an approved fallback.
• Withdrawn, contradictory, or badly extracted content can drive consequential answers.
• Independent evaluation is compromised, critical security findings are open, or no one owns incidents.

A blocker pauses that source or feature; it does not cancel the project. A conditional go suits only a constrained pilot with approved sources, users, tasks, fallback behavior, and monitoring.

Everything else needs an owner and a target date. Readiness is the first stage of enterprise AI development, not a separate project.

Next step

To see how a governed, permission-aware assistant handles content like yours, book a demo with LuMay and bring your intended workflow. You can also browse LuMay's full range of AI services.