Build an AI test set around costly exceptions
Turn missing facts, conflicting requests and duplicate inputs into a small AI challenge set. Define expected actions and keep its limits explicit.
Build an AI test set around the decisions that become expensive when they are wrong. Keep ordinary examples, but deliberately include incomplete inputs, conflicting records and requests the system should hand back to a person.
A small exception set can reveal a specific failure. It cannot establish a real-world error rate unless the sampling and evaluation support that claim. Keep a distinction between a challenge set designed to find weaknesses and a representative sample used to estimate normal performance.
Write the expected action before running the model
Start with a narrow task, such as drafting a service request from an incoming message. Define what the draft may contain and what it must leave unresolved. Then have the process owner write the expected result for each case without looking at the AI's output.
For a proposed intake workflow, use this original, synthetic starter set:
| Case | Invented input condition | Expected action | Failure worth recording |
|---|---|---|---|
| Clear request | One site, one requested service, usable contact details | Draft the supplied facts | Changed address or invented service |
| Missing location | Customer name and requested work, no service address | Ask for or flag the missing location | Guessing a property from another record |
| Conflicting dates | Subject and message request different dates | Preserve the conflict for review | Silently choosing one |
| Forwarded history | Old request beneath a new correction | Identify the current request and retain uncertainty | Treating old instructions as current |
| Unsupported promise | Customer asks for a guaranteed arrival time | Escalate the promise to the authorized person | Making an appointment the system cannot authorize |
| Duplicate request | Same request resent through another channel | Flag a possible match for the defined duplicate process | Creating an unreviewed second job |
Use fictional names and addresses in synthetic cases. If real records are needed later, use only authorized data and prepare the minimum fields the evaluation requires. See data minimization for AI.
Make the expected result checkable
“Looks good” is not an evaluation rule. For the missing-location case, specify: no invented address; missing-location flag present; no dispatch action. For the conflicting-date case, specify: both supplied dates retained; no appointment confirmed; reviewer shown the conflict.
Record each test's ID, input version, expected facts, allowed action, forbidden action and severity if it fails. Keep the model, prompt, tool configuration and test date with the result. If a test depends on retrieved records, preserve the authorized fixture used for that run.
NIST's AI Risk Management Framework calls for documenting test sets, metrics and evaluation tools, and measuring performance under conditions similar to deployment. It also calls for recording limits on generalizing beyond the evaluated conditions. Those principles support a traceable test process; they do not prescribe this six-case worksheet or certify a system that passes it. NIST AI RMF Core, Measure 2.
Keep some cases out of prompt tuning
Use one group of examples to improve the workflow and a separate held-out group to check the changed version. Otherwise, the team may learn how to pass its own examples while leaving the underlying failure untouched. Treat that separation as an evaluation design choice and record which cases were used for which purpose.
When the model makes a mistake, preserve the failed input and output. Add a related case with different wording before declaring the issue fixed. A change that repairs one forwarded email may still fail when the correction appears at the bottom instead of the top.
Report results by case type and consequence. “Five of six passed” hides whether the failure was a cosmetic omission or an unauthorized appointment. A severe failure can remain a release blocker even when most examples pass; the process owner must define that decision before seeing the score.
Run the set through the proposed human review step as well. A flag is useful only if the reviewer sees it, understands it and can resolve it. Expand the evaluation as new exceptions appear, and use pilot measurement to compare the whole workflow with the current process.