Skip to main content
RoutMe

Fundamentals

What is grounding in AI support, and how to test it

By the RoutMe team. Last updated .

Grounding means that each factual claim an AI agent makes can be traced to a source: a document you provided or a live result from a system. A grounded agent answers from those sources or says it does not know. You test it with a fixed set of questions whose correct answers you already know, including questions that have no answer.

What grounding is

A language model can produce fluent text about almost anything, including things that are not true for your business. Hallucination is the name for confident output that is not supported by facts. Grounding is the practice of tying the model's output to evidence, so that return windows, prices, delivery states and appointment times come from your data and not from the model's guesses.

Evidence comes from two places. Retrieval supplies passages from documents such as policies and help pages. Tools supply live records such as an order, a booking or an account status. RAG vs tools for customer support explains when each is the right source. A grounded agent uses the right one for the question and does not blend in remembered knowledge.

What ungrounded answers look like

  • A policy stated in general terms that is not your policy, such as an invented return period.
  • A delivery date given without a lookup.
  • A product feature claimed from general knowledge of similar products.
  • A confident answer to a question your documents never cover.
  • A reply that quotes a source but alters its meaning.

The last one is the subtle one. Citation alone is not grounding. The cited passage has to actually support the claim.

Build a grounding test set

Start with 30 to 50 questions drawn from real conversations, adjusted as your volume allows. The exact number matters less than coverage. Include these groups.

  1. Answerable from documents: the policy is clear and the right answer is known.
  2. Answerable from a tool: the answer depends on a record, and you control the test data.
  3. Unanswerable: nothing in your sources covers it. The correct behavior is to say so and hand off.
  4. Conflicting: two documents disagree, perhaps an old and a new policy. The correct behavior is to follow the current one or escalate.
  5. Tempting: questions where the model's general knowledge is plausible but wrong for you.
  6. Stale: a document that changed recently. Does the agent use the new version?

For each question, write down the expected answer in a few words and the required source. Do not write the full ideal reply, because phrasing varies and you want to test facts.

Score the results

For every answer, check three things.

CheckQuestion to ask
CorrectIs every factual claim right, according to your expected answer?
SupportedCan each claim be traced to a retrieved passage or a tool result?
Appropriate abstentionWhen the answer is not in the sources, did the agent decline and hand off?

Mark each as pass or fail. A single unsupported claim in an otherwise good reply is still a fail, because customers treat the whole reply as the business speaking. Keep a table of results over time so you can see whether a change helped or hurt.

Debug failures by layer

When a test fails, find the layer that caused it before editing the prompt.

  • Retrieval missed: the relevant passage exists but was not retrieved. Improve document structure, titles and chunking.
  • Document wrong or missing: fix the content. No model setting repairs a missing policy.
  • Tool returned too little: expand the fields the tool exposes, within your privacy limits.
  • Instructions too loose: add a clear rule such as "answer from the provided sources only, otherwise offer a handoff." The guide on what to put in an AI agent's instructions has a template.
  • Model overreached: tighten the rule, add a test and consider a handoff rule that triggers when no source is found.

Live lookups reduce a whole class of errors

Anything that changes often should come from a tool call, not from a document. Order state, stock and appointment slots go stale in a document within hours. When a tool returns the record at the moment the customer asks, the answer is as fresh as your system. That is the idea behind how an AI agent looks up order status safely.

Keep testing after launch

Grounding drifts. Documents change, products change and someone edits an instruction. Re-run the set whenever any of those happen, and add every real failure you find in production as a new test. Also sample live conversations weekly, as described in how to measure AI support quality, because real customers ask things your set never imagined.

How RoutMe approaches this

RoutMe agents answer from the knowledge sources you add and from live tool results through MCP, and they are designed to pass the conversation to a person when neither source supports an answer. Read the conversation transcript to check whether a claim had a source. We do not publish accuracy numbers, because they depend on your content and your systems. Test with your own questions before you launch. See the overview of what to look for in an AI agent for related buyer questions.

Sources

Explore the product: RoutMe for Shopify, solutions and the documentation.

Frequently asked questions

Is grounding the same as RAG?

No. RAG is one technique for supplying documents to the model. Grounding is the goal of tying answers to evidence, which can also be achieved with live tool calls.

Can grounding eliminate hallucinations?

It reduces them a great deal when sources are good, but it cannot guarantee none. That is why testing, logging and handoff rules still matter.

How often should I run the test set?

After any change to instructions, documents or tools, and on a regular schedule even when nothing changed, since models and content both drift.

What should the agent say when it has no source?

Something short and honest, along with an offer to connect the customer to a person. See when an AI agent should hand off to a human.

Put these ideas to work