Playbook
How to measure AI support quality
By the RoutMe team. Last updated .
You measure AI support quality by counting what happened to each conversation, then checking a sample by hand. The core numbers are deflection, resolution, containment and handoff rate, and each one needs a written definition before you trust it. Targets should come from your own baseline, not from a vendor chart.
Why definitions come first
Two teams can report "60% deflection" and mean different things. One counts any conversation that never reached an agent. The other counts only conversations where the customer confirmed the answer. The first number rewards an assistant that gives up politely, and the second rewards one that actually helps. Agree on wording early, put it in a shared document and keep it stable so month-over-month comparisons stay honest.
Also decide the unit. A conversation, a session and a ticket are three different things. A customer who writes on Monday and again on Thursday may be one ticket, two conversations or several sessions. Pick one unit and use it everywhere.
The four core metrics
| Metric | Formula | What it tells you | Common trap |
|---|---|---|---|
| Containment rate | Conversations that ended with no human involvement / all conversations | How often the assistant stayed in charge | Counts abandoned chats as success |
| Deflection rate | Contacts that would have become tickets but did not / expected tickets | Whether volume moved away from staff | Needs an honest estimate of the counterfactual |
| Resolution rate | Conversations where the customer's need was met / all conversations | Whether people got what they came for | Hard to detect without a signal |
| Handoff rate | Conversations passed to a person / all conversations | How often the assistant stepped aside | Low is not automatically good |
Containment is the easiest to compute, because it only needs a flag for whether a person joined. Deflection is softer. You estimate what would have happened without the assistant, usually by comparing ticket volume for the same topics before and after launch, while accounting for seasonality and campaigns.
Resolution is the one that matters most and the one that is hardest to observe. You can infer it from signals: the customer clicked a thumbs-up, did not return on the same topic within a window you choose, or confirmed in words that the issue was fixed. None of these is perfect. Combine them and label the result as an estimate.
Handoff rate is a dial
A handoff is not a failure. When a customer asks for a person, when a payment dispute appears or when the assistant lacks a source, passing the conversation on is the correct behavior. Split handoffs into reasons: customer request, policy boundary, sensitive topic, low confidence, repeated failure. A rising count under "low confidence" points at missing knowledge. A rising count under "customer request" right after the first reply points at tone or at a clumsy opening.
If you push the overall handoff rate down by hiding the option to reach a person, containment improves and trust drops. Our guide to when an AI agent should hand off to a human covers the triggers worth keeping, and human handoff best practices covers the experience after the pass.
Add accuracy and safety checks
Volume metrics say nothing about whether answers were right. Add two further measures.
- Answer accuracy: of the sampled conversations, how many contained only correct statements, checked against your systems and policies.
- Policy violations: how many replies promised something the business does not offer, such as an exception, a refund or a delivery date.
Both come from reviewing transcripts. For the accuracy side, see what grounding means and how to test it, which describes a repeatable question set you can run after every change.
Sample conversations on a schedule
Dashboards compress. Transcripts explain. Every week, pull a random set of conversations, stratified so you see some from each channel and each outcome: contained, handed off, abandoned. Read them as a customer would. Mark each one against a short rubric:
- Did the assistant understand the request?
- Was every factual claim supported by a lookup or a document?
- Was the tone appropriate?
- If it handed off, was the context complete?
- Would you be comfortable if this conversation were quoted back to you?
Keep the rubric stable. Rotate reviewers occasionally so one person's habits do not become the standard. Save the failures as test cases, because a bug found once should be caught automatically the next time.
Capture customer signals
Quantitative signals from customers help, with caveats. A short rating after the conversation is easy to collect but skews toward people with strong feelings. Repeat contacts on the same topic within a short window you define are a decent proxy for unresolved issues. Follow-up messages that begin with "no" or "that is not what I asked" are worth counting, because they mark a miss even when nobody rated anything.
For order-status conversations, you can also compare what the assistant said with what the system held at that moment. That check is mechanical, and how an AI agent looks up order status safely explains why live lookups make it possible.
Set targets from your own baseline
Before launch, record your current ticket volume by topic, first response time and resolution time. After launch, compare topic by topic. A topic such as order status may move a lot because the answer is a lookup, while a topic such as a complaint should still reach people. Targets should differ per topic. Resist any single blended target, because it hides the topics where the assistant is weak.
Avoid quoting someone else's percentages in your internal planning. Their catalog, customers and definitions differ from yours. Your own baseline, measured the same way before and after, is the only comparison that holds up.
How RoutMe approaches this
RoutMe keeps the whole thread and each tool call, so a reviewer can see what was looked up and what the assistant said, and handed-off conversations arrive in a shared inbox with full context. That gives you the raw material for sampling and for splitting handoffs by reason. The support deflection and human handoff solution describes the workflow, and pricing explains what is billed. Metric definitions stay yours: use the wording from this guide or your own, but write it down.
Sources
Explore the product: RoutMe for Shopify, solutions and the documentation.