Every AI translation demo looks good. The vendor picks clean source text, a common language pair and a short file. Your real content is messier: product names, regulated phrases, tables, and languages your team rarely sees.
This guide gives procurement, localization and engineering leads a way to compare platforms on what actually breaks at scale. It ends with a pilot design you can run in a few weeks.
Step 1: Sort Your Content by Risk
Not every document deserves the same workflow. Before you look at vendors, group content into tiers by the harm a bad translation could cause.
● Tier A, human review required: contracts, regulatory filings, safety labels, patient or customer instructions
● Tier B, AI plus targeted review: support articles, product UI, internal policies
● Tier C, AI only: internal notes, research summaries, low-traffic pages
The tiers drive everything else. A platform that cannot route Tier A content to a named reviewer is not an enterprise option, however fluent its output.
Step 2: Map Volume, Formats and Languages
Write down what you translate today and what you expect in 18 months.
● Monthly word volume by tier
● File types: DOCX, PDF, XLIFF, JSON strings, slides, scanned documents
● Language pairs, including any low-resource languages
● Turnaround expectations by content type
Scanned PDFs and complex tables are common failure points. Include them in testing even if they are a small share of volume.
Step 3: Check Terminology Control
Terminology errors cause most brand and compliance complaints. Ask how glossaries are applied: during translation, or as suggestions after the fact.
● Shared term bases across teams and projects
● Version history on glossary changes, with approver names
● Do-not-translate lists for product names and legal terms
● Reporting on where glossary terms were overridden
Step 4: Ask How Quality Is Scored
"High accuracy" means nothing without a method. Ask whether the vendor reports quality using a published framework such as Multidimensional Quality Metrics (MQM), which classifies errors by type and severity.
● Segment-level quality estimates, not only document averages
● Automatic routing of low-confidence segments to reviewers
● The ability to set different thresholds per tier
Step 5: Require an Audit Trail
For regulated content, you need to show who translated what, with which engine, glossary version and reviewer. That record should be exportable.
● Source and target versions stored together
● Engine or model version captured per job
● Reviewer identity, timestamp and decision
● Retention period you control
Audit needs grow as AI enters regulated workflows. The NIST AI Risk Management Framework gives useful language for documenting how AI output is governed.
Step 6: Review Data Handling
Ask direct questions and get written answers before a pilot.
● Is our content used to train any model, including the vendor's own?
● Where is data processed and stored?
● How long are files and logs retained, and can we delete them?
● Which subprocessors handle our content?
● Is single sign-on and role-based access available?
Ask for current compliance documents rather than relying on badges on a website.
Step 7: Test Integrations, Not Connector Lists
A connector listed on a slide is not a working integration. Test the specific path your content takes.
● Content management system or repository you use today
● Two-way sync, so translated files return without manual upload
● API access, with documentation your engineers can read
● Behavior when the source file changes mid-job
Character encoding and right-to-left scripts need checking too. The W3C Internationalization resources are a reliable reference for what correct handling looks like.
Step 8: Model the Full Cost
License price is only part of the cost. Include review time, project management and rework.
● Platform fees and any per-word or per-character charges
● Reviewer hours by tier, before and after AI
● Integration build and maintenance
● Cost of errors caught late, such as reprinted labels
Scoring Sheet
Weight each area to your risk profile. A legal team will weight audit higher than a marketing team.
| Area | What good looks like | Suggested weight |
|---|---|---|
| Terminology | Enforced during translation, versioned | High |
| Quality scoring | Segment-level, MQM-style categories | High |
| Review routing | Rules by tier, language and confidence | High |
| Audit trail | Exportable, per job and reviewer | High for regulated |
| Data handling | Written no-training terms, deletion control | High |
| Integrations | Tested two-way sync with your systems | Medium |
| Cost | Clear total including review effort | Medium |
Step 9: Run a Fair Pilot
Give every finalist the same content and the same rules. Two to four weeks is usually enough.
- Pick 5,000 to 10,000 words per vendor across all three tiers
- Include your three most important language pairs and one difficult one
- Seed the source with glossary terms and a few known traps
- Have the same reviewers score every output, without knowing the vendor
- Time the full workflow, from upload to approved file
Example: a medical device firm pilots three platforms on an instructions-for-use document, a support article and a UI string file. Reviewers log every terminology miss. One vendor produces the most fluent text but ignores the glossary on 6 of 40 controlled terms. It drops out.
Questions to Ask Every Vendor
● Show us a full audit record for one translated document
● How do you stop an unreviewed Tier A file from being published?
● What happens to our glossary and memory if we leave?
● How are model updates announced and tested?
Plan the Rollout After Selection
Choosing the platform is half the work. Rollout decides whether teams use it.
● Start with one Tier B content type, such as support articles
● Load approved glossaries and translation memory before the first job
● Train reviewers on the quality categories you will use
● Review scores weekly for the first two months
● Add Tier A content only after the review workflow runs smoothly
Keep a short log of terminology changes and reviewer feedback. It becomes your evidence when expanding to new languages or teams.
Our Own Product
Disclosure: LuMay offers AI Translation Agents, which describe glossary tuning, a certified review workflow and audit traceability. Score it with the same sheet as any other platform. For custom builds, see LuMay's AI engineering services.
Next Step
Sort a month of real content into tiers this week. That one exercise will tell you which features are essential. For help designing a pilot, talk to the LuMay team.
References
● MQM Council, Multidimensional Quality Metrics: https://themqm.org/
● NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
● W3C Internationalization: https://www.w3.org/International/
