Skip to main content
RoutMe
All posts
Playbooks3 min read

Decision trees don't survive contact with customers

Every scripted bot works in the demo and falls over in production. The failure is structural, not a matter of adding more branches, and the fix is narrower than replacing the whole thing with an LLM.

VR

Vinita Rathi

Product and conversation design

The demo always goes well. Someone types 'where is my order', the bot asks for an order ID, looks it up, and reports that it is out for delivery. Everyone agrees this will deflect a third of the support queue.

Then it ships, and the first real message is: 'hey so i ordered two things last week and one showed up but the other one says delivered and it definitely isnt here, also can i still change the address on the second one'.

Why more branches never fix it

A decision tree encodes the questions you anticipated. Real messages are compound, ambiguous, out of order, and frequently contain a correction halfway through. Each of those properties multiplies the branches you would need, and they multiply against each other rather than adding.

So the tree grows, and the growth has a characteristic shape that anyone who has maintained one will recognise:

  • A catch-all fallback that quietly becomes the most-hit node in the flow.
  • Keyword matching bolted onto intent detection, because intent detection kept missing.
  • A 'talk to a human' escape hatch that customers learn to trigger immediately, defeating the point.
  • Nobody willing to touch the flow after the person who built it left.
The tree is not failing because it is too small. It is failing because it has to enumerate a space that cannot be enumerated.

The overcorrection

The obvious response is to throw the tree away and let a language model handle the whole conversation. This trades a predictable failure for an unpredictable one. A model with no constraints will happily invent a refund policy, promise a delivery date it cannot know, or agree to a discount you never authorised. In a support context those are not amusing hallucinations, they are commitments a customer will hold you to.

You also lose the thing the tree was good at. Deterministic steps are genuinely correct for deterministic problems. Sending an OTP, checking a payment status, routing by language, applying business hours, none of these benefit from judgement.

Put the judgement in one node

The structure that holds up in production keeps the flow deterministic and confines the open-ended reasoning to a single step, with explicit tools and an explicit boundary.

  1. 1Triggers and routing stay rule-based. You know what a keyword match is worth and it costs nothing to be certain about it.
  2. 2The agent node receives the customer's actual message along with a defined set of tools, read the order, check the refund window, look up the account. It decides which to call and in what order.
  3. 3Everything the agent can reach is something you explicitly handed it. It cannot query a table you did not expose or take an action you did not define.
  4. 4The branch after the agent is deterministic again: resolved, needs a human, needs a follow-up. The agent classifies; the flow decides what happens.

The compound message above stops being a problem in this shape. The agent reads it, calls the order lookup twice, sees one delivered and one in transit, answers both parts, and finds the address-change window is closed on the second item, so it routes to a human with the full transcript rather than guessing.

What to measure instead of deflection rate

Deflection rate rewards a bot for ending conversations, which is the wrong incentive, a bot that stonewalls deflects beautifully. Better signals:

  • Turns to resolution. Directly tied to cost now that service messages are billed per message, and it falls when the agent can read data instead of interrogating the customer.
  • Handoff quality. When a human picks up, do they have the transcript, the tool calls, and the account context, or do they start from nothing?
  • Repeat contact within 48 hours. The honest measure of whether the first answer was actually an answer.
  • Tool call failure rate. Usually the real cause when an agent starts behaving strangely.

None of this requires believing an AI agent will resolve everything. It requires being precise about which part of the conversation genuinely needs judgement, and refusing to let that part expand into the rest of the system.

  • Automation
  • Support
  • Design
VR

Vinita Rathi

Product and conversation design

Writes about where automation belongs in a support conversation, and where it does not.

See it on your own data

Connect a system, drop an agent node into a flow, and watch it answer from what is actually true in your systems right now.