“The AI is 98% accurate.” Before accepting that result, ask what the remaining 2% contains. A missed urgent request and a routine message sent to the wrong queue may count as one error each, yet create very different consequences.
For a leader, the useful question is whether the system makes acceptable mistakes, on relevant work, under conditions the organization can manage. A headline score is a starting point for that conversation.
My VORTA material connects success criteria, risk appetite, test plans and accountable ownership. Applying that logic to an AI evaluation means translating a technical score into a decision about the actual work. Here is a practical way to do that without turning the meeting into a statistics lecture.
Start with the decision the score is meant to support
Specify the action being evaluated. “Understand customer messages” is too broad. “Identify messages that need urgent escalation before the next routine review” gives the team something concrete to test.
Then establish who decides what the correct outcome should be. If experienced staff disagree about urgency, resolve the policy and labelling guidance first. Otherwise, an accuracy score can hide disagreement about the task itself.
The NIST AI Risk Management Framework 1.0 discussion of validity and reliability calls for accuracy measures to be accompanied by realistic, representative test sets and documented methods. It also highlights different error types and the role of human judgment in choosing metrics and thresholds. Those principles are useful when reviewing an internal prototype as well as an external product.
Two systems can have the same accuracy and different value
Consider this deliberately simplified, illustrative example. It is not a client result or a recommended acceptance threshold.
A test contains 1,000 messages. Staff have classified 20 as urgent and 980 as routine.
- System A marks every message as routine. It gets 980 classifications right and reports 98% accuracy. It misses all 20 urgent messages.
- System B identifies 18 of the 20 urgent messages. It misses two and incorrectly flags 18 routine messages. It also gets 980 classifications right: the same 98% accuracy.
System B finds much more of the work the team cares about, but creates a queue of 36 flagged messages, half of which are false alarms. That review effort matters. So do the two missed urgent cases. Neither the headline accuracy nor this example alone establishes that the system is safe to deploy.
Google’s explanation of classification metrics distinguishes accuracy from precision and recall. In everyday terms: precision asks what proportion of flagged cases really belong in that category; recall asks what proportion of the relevant cases the system found. In this example, System B has 50% precision and 90% recall for urgent messages.
The management decision is which errors the organization can tolerate and what it will do about them. A team with limited review capacity may struggle with false alarms. A service with severe consequences from missed urgent requests may prioritize reducing those misses, while still checking whether the resulting review queue is workable.
Ask for the mistakes, not just their average
Request a short error review with the process owner. Read actual failed test cases, using appropriately protected or synthetic data. Group them by business consequence rather than treating every failure as interchangeable.
For the message-routing example, useful groups could include:
- Urgent messages wrongly left in a routine queue.
- Routine messages unnecessarily escalated.
- Messages with missing context that should be referred for clarification.
- Inputs outside the agreed scope, such as an unsupported language or an unreadable attachment.
Separate frequency from severity. A common inconvenience and a rare, serious failure need different responses. Also ask whether reviewers could realistically notice the error: a visible false alarm is easier to examine than a missed case that never enters the review queue.
Human review should be tested as part of the process. It does not automatically remove the risk simply because someone is named as a reviewer. The practical conditions for intervention are covered in my article on human oversight.
Check what the test leaves out
A result on clean, familiar examples may tell you little about everyday work with incomplete records, short messages or conflicting information. Ask which conditions were included and which remain untested.
Look at relevant segments separately, such as language, request type and input quality. Record the number of examples in each segment. A high percentage based on a handful of cases is weak evidence; an empty segment is an evidence gap. Avoid collecting sensitive personal attributes merely to make the report look more detailed.
Keep a representative test set separate from a deliberately difficult challenge set. The first helps estimate performance on the intended workload. The second probes important failure modes. Combining them without explaining the mixture can make the overall score difficult to interpret.
Ask whether the evaluation examples were kept separate from the examples used to tune prompts, routing rules or models. If the same cases are repeatedly used to improve the system, success on those cases becomes less convincing evidence of performance on new work. Preserve a fresh check before making the operational decision.
Translate the evidence into a short decision record
The technical team should explain the measurements. The business owner should help determine what they mean for the service. Bring the two together in a record containing:
- Intended use: the specific task, users and operating conditions covered.
- Consequential errors: which failures matter most and why.
- Evidence: test scope, case counts, error breakdown and known limitations.
- Operating response: review arrangements, handling of uncertain cases and escalation responsibilities.
- Decision: proceed within a defined scope, narrow the scope, improve and retest, or defer.
Set acceptance criteria before looking at the final results. Include review workload and the ability to meet service commitments, not only model scores. If the evidence is too thin for a consequential decision, the next step may be more testing rather than a more confident presentation.
For generative tasks, a classification score may not fit at all. A drafted response might be factually correct but incomplete, unsupported by its sources or inappropriate for the recipient. Define a task-specific review rubric and test those qualities separately. Do not label a subjective overall impression as a universally comparable accuracy figure.
Finally, connect quality to the completed job. Include time spent reviewing, correcting and escalating outputs when measuring business value. Revisit the evidence when the task, data or system changes through an appropriate change-control review.
A useful evaluation leaves a leader able to explain what was tested, which failures remain and why the proposed scope is justified. If those answers are missing, another decimal place in the accuracy score will not resolve the decision.
If you are reviewing an AI proposal or deciding what evidence your team needs, schedule a consultation to connect the evaluation to the business process and its risks.