Most companies can tell you how many employees finished their AI course. Far fewer can tell you whether anyone works differently a month later. That gap turns renewal decisions into guesswork.
This guide gives engineering leaders, operations managers and legal teams a practical way to measure AI training. It covers what to record before the first session, how to test real skill, and how to decide whether to scale, fix or stop.
Why Standard Training Metrics Fall Short
Classic training evaluation, such as the Kirkpatrick Model, moves from reaction to learning, behavior and results. Most AI programs stop at the first two: a satisfaction survey and a quiz.
AI work adds a twist. The core skill is judgment: spotting when a confident answer is wrong. Quizzes rarely test that, so a scorecard has to.
Start With One Task, Not "AI Skills"
"Improve AI skills" cannot be measured. A named, recurring task can. Before you buy or launch training, choose one or two tasks per team that the program should change.
● Engineering: writing unit tests for an existing service with an approved coding assistant
● Operations: drafting the weekly exceptions summary from a ticket export
● Legal: producing a first-pass issues list for a standard vendor contract
Each task should recur often enough to measure within weeks. It also needs an owner who already reviews the output.
Record a Baseline Before Training Starts
Without a baseline, any later number is a story, not evidence. Capture the current state for each chosen task in the weeks before training.
● Time: typical minutes from start to an approved result, counting rework
● Quality: reviewer score against a short written rubric
● Risk: policy or data-handling issues found in review
○ Note tools, staffing and volume during the baseline period
If those conditions change later, write it down. A new tool release or a quiet month can move numbers more than the course did.
Write a One-Page Rubric
Reviewers need the same yardstick before and after. Keep the rubric to four or five criteria, each scored 1 to 4.
| Criterion | 1 | 4 |
|---|---|---|
| Accuracy | Errors reach the final output | No errors in final output |
| Verification | No checks against source | Every claim checked against source |
| Policy | Unapproved tool or data used | Approved tool, correct data handling |
| Escalation | Risky items not flagged | Risky items flagged to the right owner |
Use the same reviewers for baseline and follow-up where you can. It reduces scoring drift.
Test Skill With Work Samples, Not Quizzes
At the end of training, give each person a realistic task and an AI-generated draft containing a few planted problems.
➤ Engineering: a generated test file that passes but never checks the failure path
➤ Operations: a summary that misstates one figure from the source export
➤ Legal: an issues list that misses an indemnity clause
Score whether they caught the problem, fixed it and escalated where policy requires. This is the closest proxy for real-world judgment.
Check Behavior at 30 and 60 Days
Skills fade when deadlines return. Schedule follow-ups about one and two months after training. Ask managers and reviewers, not only learners.
● Are people using the approved tool for the chosen task?
● Are required review steps being followed?
● Has anyone switched to an unapproved tool?
○ Pull usage data where tools provide it, and pair it with reviewed samples
Low use is a signal, not a verdict. It often points to missing access, unclear policy or a workflow that was never updated.
Compare Results Against the Baseline
Rerun the baseline measures on the same task. Read time and quality together. Faster output that needs more rework is not an improvement.
| Evidence type | What it shows | What it cannot show |
|---|---|---|
| Completion and satisfaction | Reach and experience | Skill or business value |
| Work-sample score | Ability under realistic conditions | Use on the job |
| 30/60-day usage and review | Changed behavior | Cause of the change |
| Baseline comparison | Change in the task | Certainty, if other conditions moved |
Where practical, train one group first and compare it with a group that has not started. A staged rollout gives a fairer read than a single before-and-after chart.
The Team Scorecard
Bring the measures together on one page per team. Leaders should be able to read it in two minutes.
| Measure | Engineering | Operations | Legal |
|---|---|---|---|
| Chosen task | Unit tests | Exceptions summary | Contract issues list |
| Baseline time | Record | Record | Record |
| Post-training time (60 days) | Record | Record | Record |
| Rubric score, before and after | Record | Record | Record |
| Work-sample catch rate | Record | Record | Record |
| Approved-tool use at 60 days | Record | Record | Record |
| Decision | Scale, fix or stop | Scale, fix or stop | Scale, fix or stop |
Add one shared row for responsible use across every team, such as policy issues per 100 reviewed outputs.
Worked Example
A finance-operations team picks the weekly exceptions summary. Baseline: around 90 minutes with two rework rounds, rubric average 2.4. After training, the work sample shows 11 of 14 people caught the planted figure error.
At 60 days, the summary takes about 50 minutes with one rework round, and the rubric average is 3.3. Usage is high. The decision is to scale the program to the next team. These figures are illustrative.
Decide: Scale, Fix or Stop
Agree on decision rules before results arrive, so the data drives the call rather than enthusiasm.
➤ Scale when work-sample scores, usage and task quality all moved the right way
➤ Fix when skill is high but usage is low; look at access, policy and workflow first
➤ Stop or redesign when skill did not improve; the content or format likely missed the task
Share the scorecard with the training provider. A good partner should welcome it and adjust the program.
Link the Scorecard to Risk Management
AI training is also a control. Guidance like the NIST AI Risk Management Framework and its Playbook treat staff competence as part of governing AI. A scorecard gives you evidence for that.
Questions to Ask a Training Provider
● Will you help us define tasks and baselines before the program starts?
● Do participants do hands-on work on our real use cases?
● How do you assess judgment, not just completion?
● What follow-up do you offer after the final session?
For program comparisons, see our guides to AI training for employees and AI training for executives.
Common Measurement Mistakes
● Measuring too many tasks at once, so nothing gets tracked well
● Changing reviewers between baseline and follow-up
● Counting logins as adoption
● Reading results before 30 days, when novelty inflates usage
Our Own Product
Disclosure: LuMay offers role-based AI training through LuMay Academy and hands-on AI Learning Labs. Use the questions above to evaluate any provider, including us.
Measure What Changes the Work
Training is worth renewing when the chosen tasks get better and stay better. Pick the task, record the baseline, test judgment and check back. For help building a role-based program, request an Academy engagement.
References
● Kirkpatrick Partners, The Kirkpatrick Model: https://www.kirkpatrickpartners.com/the-kirkpatrick-model/
● NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
● NIST AI RMF Playbook: https://airc.nist.gov/airmf-resources/playbook/
