AI call summaries for HVAC: Test what the dispatcher must not lose
Test an AI call summary against caller facts, uncertainty and missing dispatch information. Includes synthetic cases and a reviewer correction log.
Test an AI call summary by asking whether a dispatcher can distinguish what the caller said, what remains unknown and what a person must decide. A fluent paragraph that turns uncertainty into a confirmed diagnosis is a poor dispatch record.
Begin with draft summaries reviewed by the office. Use synthetic calls or recordings you are authorized to process, with the relevant data and recording requirements already resolved. The test below concerns preservation of information; it is not a diagnostic method or an emergency-response script.
Give the summary a limited job
The draft should capture the caller's reported issue, the relevant location or equipment identifier supplied, timing, availability, prior visit references and unanswered questions. It should not invent a cause, promise an arrival time, choose a repair or silently assign urgency.
Have the service manager specify the company's existing escalation rules and who applies them. If a call raises a condition outside the pilot's permitted scope, route it to that person. Do not ask the model to create a new safety policy from the conversation.
This approach follows NIST's emphasis on understanding the context of use, measuring behavior and managing identified risks. The specific cases here are original DATUM test material, not a NIST-certified assessment. NIST AI RMF Core
Test changes of mind, uncertainty and missing facts
These invented excerpts are deliberately short. They are not recordings of customers.
| Synthetic caller excerpt | What the draft should preserve | Unsupported rewrite |
|---|---|---|
| “The upstairs unit stopped yesterday. Actually, it started again this morning, but the noise is still there.” | Reported interruption, later restart and continuing noise | “Unit is currently not running” |
| “I think it is the unit labeled B, but I am not at the building.” | Equipment identity is unconfirmed | “Service unit B” as a confirmed instruction |
| “We can provide access after two. I am asking whether tomorrow is possible.” | Access constraint and requested day, both subject to scheduling | “Appointment confirmed tomorrow at two” |
| “Someone came last month. I do not have the job number.” | Prior visit reported; reference missing | An invented job number or invented previous repair |
| “It is the rear office, not the front office.” | Corrected location | Retaining the first location alone |
A summary can be concise and still carry the correction. Require a clear distinction between a reported symptom and an established fact about the equipment. Retain the source reference so a reviewer can inspect the relevant statement when necessary.
Review the output by field, not by tone
Create a review row for each required fact: source phrase or timestamp, proposed field, draft value, reviewer decision and correction. Use “accurate,” “missing,” “unsupported,” or “uncertainty lost.” A single summary may have more than one issue.
For example, the invented second call should produce something like: “Caller believes the equipment may be unit B; identity needs confirmation.” A reviewer should reject “Unit B requires repair” because both the identity and required action have become more certain than the source supports.
Do not accept the draft merely because it contains all the expected field names. An empty equipment-ID field can be correct when the source does not establish the identifier. An invented value is worse than an explicit request for clarification.
Measure the dispatch work the draft creates
Time the current manual summary and the full review of the AI draft on comparable permitted cases. Count corrections, source replays and follow-up questions. A faster initial draft is not a saving if the dispatcher spends more time checking it.
Keep errors that could change the dispatch decision visible separately from spelling or phrasing edits. Set the acceptance and stop rules with the service manager before the test. Do not use an attractive average score to conceal one kind of consequential mistake.
The AI pilot measurement worksheet helps include preparation, review and exception work. If you are also evaluating automated call handling, the contractor receptionist pilot addresses that wider boundary. A summary-only test does not establish that an AI should book, transfer or respond to calls.
Save the cases that caused a bad summary
When a draft loses a correction or invents a fact, preserve the synthetic or appropriately controlled test case and expected result. Change the prompt, input format or scope, then rerun that case alongside ordinary calls. Record the version that was tested.
Release the summary workflow only within the boundary actually evaluated. The dispatcher should receive a reviewable record, an accessible source and a visible unresolved question when the call does not establish the answer.