To choose an AI development company, score each candidate on evidence, not presentations: production systems they have shipped, how they evaluate accuracy, how they secure data and tools, who owns the code, and what a fixed-scope first phase costs.
A weighted scorecard makes those comparisons consistent.
This guide is for CIOs, CTOs and procurement leads running a shortlist of two to five vendors. It gives a 20-point scorecard, the evidence to request, red flags and a worked scoring example.
Start With Your Own Scope
Vendors cannot be compared until the problem is fixed. Before the first call, write down:
• The workflow: what task, for whom, and what changes if it works
• Systems involved: which applications and data sources the solution must read or write
• Constraints: data residency, deployment model (SaaS, your cloud, on-premises), security review requirements
• Success measure: one or two metrics agreed with the business owner
• Budget band: a range for a first phase, so proposals are comparable
Send the same one-page scope to every vendor.
The 20-Point Scorecard
Score each criterion 0–5. Multiply by the weight. Weights below are a starting point; adjust them to your priorities before scoring.
| # | Criterion | Weight | Evidence to request |
|---|---|---|---|
| 1 | Production systems in a similar domain | 3 | Named references you can call |
| 2 | Architecture proposal for your scope | 3 | Written design with trade-offs |
| 3 | Evaluation method | 3 | Sample test set and metrics |
| 4 | Data handling and residency | 3 | Data-flow diagram |
| 5 | Tool and integration security | 3 | Permission model for actions |
| 6 | Integration experience with your systems | 2 | Prior connectors or APIs used |
| 7 | Deployment options | 2 | SaaS, your cloud, on-premises |
| 8 | Monitoring and incident response | 2 | Runbook or dashboard sample |
| 9 | Human review design | 2 | Approval and escalation flow |
| 10 | Code, model and data ownership | 3 | Contract clause |
| 11 | Exit and handover plan | 2 | Handover checklist |
| 12 | Fixed-scope first phase | 2 | Priced discovery or pilot |
| 13 | Transparent pricing | 2 | Line-item estimate |
| 14 | Run-cost forecast | 2 | Monthly cost at your volume |
| 15 | Team seniority and continuity | 2 | Named team, CVs |
| 16 | Delivery cadence | 1 | Sprint plan, demo schedule |
| 17 | Governance alignment | 2 | Mapping to NIST AI RMF or your policy |
| 18 | Security posture evidence | 2 | Questionnaire responses, reports they can share |
| 19 | Change management and training | 1 | Rollout plan |
| 20 | Communication and fit | 1 | Reference feedback |
The maximum score is 5 × the sum of weights (43) = 215. Normalize to 100 for easier reading.
Questions That Separate Strong Vendors
• "Show us how you measured accuracy on a past project." Strong answers include a test set, metrics and how failures were fixed
• "What would you not build with an LLM here?" Good engineers name parts better handled by rules or search
• "What can the agent do without a human?" Look for explicit permission limits and approval gates
• "What does it cost to run at our volume?" Expect a model with assumptions, not "it depends"
• "What do we own at the end?" Code, prompts, evaluation data and infrastructure-as-code should be yours
For governance framing, the NIST AI Risk Management Framework and the OWASP Top 10 for LLM Applications give shared vocabulary for risk and security discussions.
Worked Example: Scoring Three Vendors
Illustrative example. Vendors A, B and C are hypothetical. Scores are invented to demonstrate the method.
| Group | Weight total | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Delivery evidence (1–3) | 9 | 36 | 27 | 41 |
| Security and data (4–5, 18) | 8 | 24 | 36 | 32 |
| Engineering and ops (6–9) | 8 | 28 | 21 | 28 |
| Ownership and exit (10–11) | 5 | 25 | 10 | 20 |
| Commercials (12–14) | 6 | 18 | 24 | 24 |
| Team, governance, fit (15–17, 19–20) | 7 | 28 | 32 | 32 |
| Total (of 215) | 43 | 159 | 150 | 177 |
| Normalized (of 100) | 74 | 70 | 82 |
Vendor C leads, but the buyer also checks for any criterion scored 0 or 1 on a weight-3 item. A low score there can be a deal-breaker regardless of the total.
Red Flags
• Accuracy promised before seeing your data
• No written evaluation plan or "we test it manually"
• Agents with broad credentials and no approval design
• Ownership retained by the vendor for code or prompts you paid for
• Only demos, no production references
• A single price with no assumptions
Contract Terms to Check
• Acceptance criteria tied to the agreed evaluation set
• IP assignment for deliverables, including prompts and evaluation data
• Data use: your data is not used to train shared models without consent
• Security obligations and incident notification timelines
• Handover of documentation, runbooks and infrastructure-as-code
Limitations of Scorecards
• Weights are judgment calls: two buyers can rank the same vendors differently
• Evidence quality varies: references and case studies are self-selected
• A scorecard does not replace a paid first phase: a fixed-scope discovery or pilot is the most reliable test of delivery
• Industry-specific rules apply: regulated sectors add requirements, as covered in how banks choose an AI development partner
Our own service (disclosure: LuMay is an AI development company). LuMay's pricing page lists a two-week Discovery Sprint at $10K–$15K, a 4–8 week Production Pilot at $25K–$75K, and an AI Engineering Pod at $35K–$75K per month.
We are one of the vendors you might score; apply this scorecard to us the same way.
Next Step
Test a vendor on a real, bounded problem before a large commitment. Book a two-week Discovery Sprint with LuMay and score the output against this scorecard.
Sources
• NIST AI Risk Management Framework
• OWASP Top 10 for LLM Applications
• Soca AI: Enterprise AI evaluation checklist
