Imagine an AI agent tells a customer, “You can return the product within 60 days.” Your policy says 30. The customer says “great, thanks” and leaves. No escalation, no bad CSAT, no handoff. That can look like a successful resolution (a hypothetical example; see why containment rate misleads). It’s also completely wrong.
Rippit audits third-party AI agents independently: it scores every AI-agent conversation against the same standards as your human reps, flags off-policy answers and bad handoffs, and shows which customer questions each AI agent should and shouldn't handle, whichever vendor built it, so you know what each AI agent is actually worth.
Key takeaways
- Containment and resolution metrics usually miss hallucinations, because the conversation looks resolved.
- Five checkable types cover most of them: invented policy, wrong figure, nonexistent feature or step, fabricated status and misapplied information.
- Customer corrections and repeat contacts show where to start, but the worst hallucinations are the ones customers believe, so check every conversation.
- Price them in repeat contacts, credits and lost customers, then fix the sources.
What counts as a hallucination in customer support?
Not every bad AI response is a hallucination. “I’m unable to help with that” is unhelpful, but invents nothing. The dangerous failure is a confident, specific claim your sources don’t support, in five categories:
| Type | What it looks like | The check that catches it |
|---|---|---|
| Invented policy | A refund window, eligibility rule or exception that doesn’t exist | Compare the stated rule with your current policy |
| Wrong figure | An outdated price, fee, SLA, delivery window or account limit | Compare every number with your current source |
| Nonexistent feature or step | A button, setting or feature that doesn’t exist | Check the steps against current product documentation |
| Fabricated status | A claim about the customer’s order or account it couldn’t know | Check whether the agent had access to that record |
| Misapplied information | Real information applied to the wrong product, plan, geography or customer | Check whether the source covers this customer |
Each is checkable: you don’t have to guess whether a response “sounds good.” You can test whether it’s true.
Which signals point you to the conversations worth reading?
Don’t just look for angry customers. Four signals show where to start:
- Correction language: “That’s not right.” “That’s not what your website says.”
- A repeat contact on the same problem within a window you set, such as 72 hours.
- An escalation right after a confident answer (our chatbot escalation audit scores these handoffs).
- A specific number or rule with nothing behind it: not proof, but the right pile to read first.
But the worst hallucinations may be the ones the customer believes. A customer who accepts a confident wrong answer leaves no negative signal. So don’t monitor only escalations, repeat contacts or low CSAT: look at the claims the AI agent makes.
How do you check an answer against your own source of truth?
For every AI-agent conversation: did the agent state a policy, quote a number, give instructions or make a claim about this customer’s account? If so, what information supports it? You’re evaluating whether it was true, not whether it sounded helpful.
One limitation: your source of truth has to be trustworthy. If your help center says 60 days when the real policy is 30, an AI agent and an automated auditor can agree and still be wrong. Sometimes monitoring exposes an AI problem, sometimes a knowledge problem; both are worth knowing. For the wider set of AI-agent checks, see the AI agent monitoring playbook.
What do hallucinations cost you?
A hallucination costs money when the customer comes back, when you honor the promise and when the customer leaves:
Cost of one month’s hallucinations = hallucinations found × [(share causing a repeat contact × cost per human-handled contact) + (share honored as a refund or credit × average credit) + (share that churn × annual revenue per customer)]
Illustrative example (not Rippit or vendor data): 500 hallucinations in a month; 40% cause a $6 repeat contact, 10% are honored as a $40 credit and 2% lose a $1,000-a-year customer. That’s 500 × ($2.40 + $4.00 + $20.00) = $13,200: $1,200 in repeat contacts and $2,000 in credits that month, plus $10,000 a year in revenue from 10 lost customers. The same audit shows where answers hold up, which is where more automation actually lowers cost to serve.
How does Rippit audit AI-agent conversations?
Rippit analyzes your AI agents’ conversations independently. Connect your conversation hubs in one click—including Zendesk, Intercom, Decagon and more (integrations, docs)—and Rippit reads every conversation, telling the AI agent’s replies apart from your human reps’.
Describe the checks in plain language: “Find every conversation where the AI agent stated a refund policy. Extract what it told the customer and compare it with our current refund policy. Flag anything that doesn’t match.” Try it in chat, then save it as an Agent App that runs hourly, daily or weekly and sends flags to Slack or email, each linked to the source conversation (monitor your AI agent).
Because Rippit doesn’t sell a customer-facing AI agent, one Rippit agent can hold AI agents from different vendors, and your human reps, to the same standard. Rippit audits; it doesn’t change your AI agent (how it works; why Rippit reads 100% of conversations).
What should you do with the hallucinations you find?
Don’t just count them. Group them by the question that caused them. You may find dozens came from one outdated help article, two sources that contradict each other, or questions the AI agent simply shouldn’t answer yet. Those patterns tell you what to fix. Then run the same audit again. The goal isn’t counting how often your AI agent was wrong. It’s making that number smaller.
Frequently asked questions
Isn’t containment or resolution rate enough?
No. They measure whether the AI agent handled a conversation without a person, not whether its answer was right. A confident wrong answer the customer accepts counts as a success in both.
Can we detect hallucinations without a labeled test set?
Yes, for stated, checkable claims: compare what the AI agent asserted with your current policy, pricing and documentation. Test sets evaluate a model before launch; this works on live conversations.
Does Rippit monitor the AI agent in real time?
No. Rippit analyzes conversations after they land and reports on your schedule; it doesn’t intervene.
Which AI agents can Rippit audit?
Those whose conversations reach Rippit through its integrations, including Ada, Decagon and Sierra, plus AI agents inside Intercom or Zendesk, such as Intercom Fin, and more on the integrations page. Human-handled conversations sit alongside them for comparison.
.webp)
