Why AI support agents hallucinate refunds, and how to build one that can't

The Suvenna team · July 14, 2026 · 7 min read

A customer emails: "Where's my order?" The AI helpdesk doesn't have the tracking data, so it replies with a confident delivery date and a tracking number that doesn't exist. Another customer asks about returns, and the bot grants a 60-day window when the store's policy says 30. The merchant finds out a week later, when the customer forwards the email back as proof.

If you've run AI support at any volume, you've seen a version of this. It's tempting to treat it as a bug that better models will eventually fix. It isn't. Fabrication is the default behavior of the technology, and the fix is architectural, not incremental. This post explains why language models make things up, why prompting them not to doesn't work, and what a system has to look like for fabrication to be impossible rather than merely unlikely.

Fabrication is the default, not a defect

A large language model is a next-token predictor. Its entire training objective is to produce the most plausible continuation of the text so far. That's the whole machine. There is no internal step where it consults a database of facts, and no internal distinction between retrieving something it knows and generating something that sounds like knowledge. Knowing and sounding right are the same operation.

This matters enormously for support. Ask a model about your return policy when it has never seen your return policy, and it will not return an error. It will produce a return policy: the most statistically typical one for a store like yours, rendered in fluent, confident prose. The model did exactly what it was built to do. Plausibility is the product. The problem is that in customer support, a plausible answer that happens to be wrong is worse than no answer, because it arrives wearing the merchant's name and reads like a commitment.

The distinction that matters is open-ended generation versus grounded retrieval. Open-ended generation asks the model "what would a good answer look like?" Grounded retrieval asks "what does this specific document say?" and only then asks the model to phrase it. Most AI helpdesks blur the two: they retrieve some context, hand it to the model, and hope the model stays inside it. Hope is doing a lot of work in that sentence.

The training-data confidence problem

There's a second, subtler force pushing models toward confident fabrication: the data they learned from.

Text on the internet is overwhelmingly written by people who believe they know the answer. People who don't know generally don't post. So the training distribution is saturated with confident assertions and nearly empty of calibrated abstention. The phrase "I'm not sure, let me check" is rare in writing relative to how often it's the correct response in life.

Instruction tuning then compounds this. Models are refined with human feedback, and human raters consistently prefer complete, helpful-sounding answers over hedges and refusals. An answer that says "I don't have enough information" scores worse than a fluent guess, even when the guess is the wrong response. Over millions of comparisons, that preference gets baked in: the model learns that confidence is what good answers look like.

The result is a system whose confidence is a property of its prose style, not of its evidence. It sounds equally certain when quoting your actual shipping policy and when inventing one. You cannot read reliability off the tone, and neither can your customers.

Why "please don't hallucinate" fails

The obvious response is to instruct the model: answer only from the provided documents, never guess, say "I don't know" when unsure. Every AI helpdesk does some version of this. It reduces fabrication. It does not eliminate it, and it can't, for three reasons.

First, the model cannot reliably introspect on whether it knows something. There is no lookup step that can fail and trigger the "I don't know" branch. The model generates the abstention the same way it generates everything else, by predicting whether abstention is the likely next text. Sometimes the likelier next text is a plausible guess.

Second, instructions decay. Long conversations, conflicting goals (be helpful, be brief, be cautious), and adversarial customer messages all dilute the system prompt's influence. A frustrated customer who insists "just tell me when it will arrive" is applying pressure the prompt has to outweigh, and it doesn't always.

Third, and most fundamentally, prompt compliance is probabilistic. Suppose your instructions work 99 percent of the time. At three thousand conversations a month, that's roughly thirty fabrications a month, each one a confident, wrong promise sent to a real customer under your brand. A safeguard that fails one time in a hundred is not a safeguard at that volume. If a behavior must never happen, it cannot be enforced at a layer whose outputs are probabilistic.

The architecture that makes fabrication harmless

The conclusion we drew building Suvenna: stop trying to make the model incapable of being wrong, and instead build a system in which a wrong model can't do damage. That means moving the guarantees out of the prompt and into the layers around the model. Four layers, specifically.

A closed knowledge source. The agent answers only from the merchant's own policies and saved macros, plus the verified customer's actual Shopify order record. If no document covers the question, there is nothing to phrase, and the answer path simply doesn't exist. Generation becomes selection and phrasing over retrieved facts rather than open-ended invention.

Identity-bound data access. Order lookups are bound to the verified customer's email at the data layer. The model cannot be talked into fetching someone else's order, because the query itself is scoped before the model is involved. A prompt injection can change what the model says; it cannot change what the model can see.

Abstain-and-escalate as a first-class outcome. "I don't know, a person will take this from here" is a designed, measured output with its own handoff flow, not a failure state to be minimized. The escalation carries the full conversation and order context, so the human picks up mid-stride instead of starting over.

Execution-layer bounds. This is the layer that matters most for money. Even if every layer above it failed and the model produced a wrong answer, that answer cannot move wrong money. Every refund, cancellation, and return is confirm-gated: the customer explicitly confirms, and risky actions queue for one-click human approval. Refund amounts are bounded by the order's actual refundable amount at the moment of execution, not at the moment of conversation. Live actions ship off per store until the merchant turns them on. And every action lands in an append-only audit log.

Here's the difference between prompt-level mitigation and architectural control, failure by failure:

| Failure mode | Prompt-level fix | Architectural control | | --- | --- | --- | | Invented policy answer | "Only answer from the docs" | Closed knowledge source: no document, no answer path | | Fabricated tracking number | "Never guess tracking info" | Answers come only from the real order record | | Wrong customer's order exposed | "Verify identity first" | Lookups bound to the verified email at the query layer | | Overconfident guess | "Say when you're unsure" | Abstain-and-escalate is a scored, routed outcome | | Wrong refund amount | "Be careful with refunds" | Amount bounded at execution, confirm gate, approval queue |

The left column is hope. The right column is enforcement. With the architectural controls in place, the model can still phrase something imperfectly, but its wrongness cannot become a policy commitment, a data leak, or a money movement.

What grounding costs you

Honesty requires saying what this approach gives up, because it does give up some things.

More escalations early. An agent that abstains when your policies don't cover a question will surface every gap in your documentation as a handoff, and in the first two weeks that means more tickets reaching your team than a freewheeling bot would allow. This is the system telling you exactly what to write down. Merchants who fill the gaps see escalations fall week over week, but the early period is real work.

Slightly less fluent replies. A grounded answer quotes and paraphrases your policy rather than composing freestyle empathetic prose, and it can read a touch plainer. We think that trade is correct for commerce. A customer would rather have a plain answer that's true than a beautiful one that isn't, and only one of those generates a chargeback.

It will never cover everything. Some questions genuinely need a human: judgment calls, exceptions, angry edge cases. A grounded agent makes that boundary visible instead of papering over it with improvisation. We consider that a feature, but it does mean the honest ceiling on automation is below 100 percent, and any vendor telling you otherwise is describing the improvisation.

Built this way from the first commit

Suvenna is this architecture, end to end: closed knowledge source, identity-bound order access, abstention as a first-class outcome, and a confirm-before-money execution layer that cannot be disabled in code. We onboard new stores in limited waves so every rollout gets real attention.

If you'd rather run support on a system that can't invent a refund than on a prompt that asks nicely, request early access. Founding members lock in 20 percent off their first year.