ANSWER

Why Chatbot Containment Rate Misleads, and What to Measure Instead

Containment tells you whether a chatbot conversation ended without a human. It does not tell you whether the customer’s problem was solved, or whether they called back, asked for a refund or cancelled.

Try Rippit free
or request demo
By Mike Nucci · Published October 2, 2026

Containment rate tells you whether a chatbot conversation ended without a human. It does not tell you whether the customer’s problem was solved. A customer whose question was answered correctly can count as contained. So can a customer who received a wrong answer, got frustrated, closed the chat, and emailed support the next morning. Only one was a success.

Containment measures the absence of a human. It does not measure the presence of an answer. Measure the customer outcome instead: confirmed resolution, unresolved abandonment, repeat contact, handoff quality, and wrong answers. Then join each outcome to the customer’s orders, refunds and plan, so you can see which bot failures end in a refund request or a cancellation.

With Rippit, an agent you set up reviews every bot, chat and human conversation you connect, so you can see where the bot resolves, escalates or gets answers wrong.

Picture a subscriber who asks your bot how to cancel an add-on. The bot explains, the chat ends, and the dashboard counts a contained conversation. Three days later the same customer calls, still being billed, and asks for a refund. The bot logged a success and the phone team logged a resolved call. Only the billing record shows what actually happened.

Key takeaways

  • Containment measures the absence of a human, not the presence of an answer, so an abandoned chat and a solved problem score identically.
  • Replace one number with five: confirmed resolution, unresolved abandonment, repeat contact rate, handoff quality and wrong-answer rate.
  • Confirmed resolution needs a window and a repeat-contact check; without both, you are measuring session endings.
  • Match repeat contacts on every channel, including phone. A “contained” chat followed by a call about the same order is a failed conversation.
  • Join outcomes to orders, refunds and cancellations to see which bot failures cost you customers, not just agent time.

What does chatbot containment rate actually measure?

Containment asks one question: did this interaction end without escalating to a human agent?

The problem comes when containment becomes a proxy for resolution.

Consider three conversations:

  • Conversation A: The bot gives the correct answer and the customer leaves.
  • Conversation B: The bot confidently gives the wrong answer and the customer accepts it.
  • Conversation C: The bot repeats the same troubleshooting steps until the customer stops responding.

All three can appear contained. Good AI-agent monitoring asks what actually happened to the customer’s issue.

BOT DASHBOARD

Conversation A

Contained

Conversation B

Contained

Conversation C

Contained

Containment: 3 of 3

CONVERSATION EVIDENCE · SAME 3 CHATS

A · Chat #31842

Confirmed resolution

Correct answer, customer confirmed it worked, no repeat contact.

B · Chat #31907

Wrong answer

Bot’s answer conflicts with current policy. The customer accepted it.

C · Chat #32015

Unresolved abandonment

“I already tried that.” Same steps repeated, then the customer stopped responding.

All three conversations count as contained; only one solved the customer’s problem.

What should you measure instead of chatbot containment? The Resolution Ledger

No single metric replaces it. You need five that describe different failure modes.

The Resolution Ledger: five metrics to report alongside containment
MetricWhat it measuresMeasurement windowFailure signal
Confirmed resolutionEvidence that the customer’s issue was resolved, with no same-issue repeat contactSession + the repeat-contact windowIssue appears unresolved or customer returns with the same problem, in any channel
Unresolved abandonmentCustomer stops engaging before there is evidence of resolution or successful handoffSessionConversation ends without evidence the issue was solved
Repeat contact rateCustomer contacts support again about the same underlying issueOften 72 hours; adapt to your businessSame customer, same issue, another conversation in any channel, including a phone call
Handoff qualityWhether escalation happened at the right time and context carried to the humanHandoffCustomer has to repeat information already given to the bot
Wrong-answer rateBot gives information that conflicts with the applicable policy, documentation, or known factsSessionMaterially incorrect or unsupported answer

Set the window by product (a weekly-use B2B tool needs longer than checkout). The important part is having a window at all. For the full method, including repeat contacts that come back by phone, see how to measure true resolution for AI agent conversations.

A resolution metric without a repeat-contact window is just a session-end counter with a better name.

Customers don’t always come back, so confirmed resolution should combine signals: the requested action was completed, the customer confirmed the fix, and no same-issue contact followed during the window. “The requested action was completed” is something you can check: the refund shows as issued, the order was changed, the add-on was removed.

Why measure the customer’s issue, not just the chatbot session?

Containment is session-centric. Customers are not.

A customer chats with the bot about a failed payment on Monday, chats again on Tuesday, and emails support on Wednesday, where a human finally fixes it. A dashboard sees three interactions; the customer had one problem that took three conversations to resolve.

The better data model is conversation → issue → outcome → repeat contact, rather than session → contained / not contained. Because customers rephrase (“My payment won’t go through,” then “Still can’t complete checkout”), matching ticket tags undercounts; analyze each conversation’s contact driver to decide whether a later one is the same problem.

At Resident, Michelle Zimmerman, Senior Quality Data Analyst, puts the stakes this way: “Understanding where the journey breaks down impacts retention, future sales, referrals, everything that matters. Rippit finally gives us that visibility.”

What should you join each chatbot outcome to?

An outcome label tells you what happened to the issue. Joined to the customer’s later contacts, orders and plan, it tells you what the failure cost and who should fix it.

Five joins that turn a chatbot outcome into a retention signal
JoinWhat it addsWhere it comes fromThe question it answers
Later contacts, every channelThe same customer’s next ticket, email or phone call about the same issue, matched by email, account or order number within your windowHelp desk, email, phone tickets and call recordsWhich “contained” chats came back, and on which channel?
Orders and refundsThe order, refund or charge the chat was about, and whether it was completedOrder and billing systemsDid the customer get what the bot said they would?
Plan and cancellationsThe customer’s plan and tenure, and whether they cancelled or downgraded after the chatBilling and subscription dataWhich bot failures come before a cancellation?
Help-center docs and release notesThe article or policy the bot answered from, its last-updated date, and the release the customer was onKnowledge base, release notesWas a wrong answer the bot’s fault or the article’s?
Handoff recordWhat the bot passed to the human, and how long the customer waitedHelp desk handoff fieldsDid the human start cold?

Start with later contacts on every channel, including phone. A chat-only dashboard can’t see the customer who gave up on the bot and called instead.

What can a 40% containment rate look like after you measure outcomes?

Consider a hypothetical example.

An AI support agent handles 1,000 conversations in a month, and its dashboard reports 40% containment: 400 conversations ended without a human. Inspect those 400 and give each one a single primary outcome, so the four counts add up to 400. Failure signals overlap in practice (a wrong answer is often followed by a repeat contact), so pick one rule for which outcome wins and apply it every time:

  • 80 ended in unresolved abandonment.
  • 55 were followed by another contact about the same issue.
  • 15 contained a materially incorrect answer.
  • 250 showed evidence of resolution and survived the repeat-contact check.

The headline metric was containment: 40% of all conversations. The more meaningful one was confirmed resolution: 25% of all conversations (250 of 1,000), which is 62.5% of the contained ones. A containment rate tells you where the conversation ended. An outcome analysis tells you what happened to the problem.

DASHBOARD HEADLINE

40%

Containment · 400 ended without a human

OUTCOME ANALYSIS

25%

Confirmed resolution · 250 of 1,000

Inside the 400 contained conversations (one primary outcome each)
OutcomeConversationsShare of contained
Confirmed resolution25062.5%
Unresolved abandonment8020%
Repeat contact, same issue5513.75%
Materially incorrect answer153.75%
A 40% containment rate becomes 25% confirmed resolution once each contained conversation is checked for its outcome.

What the weekly outcome report should show

The ledger counts outcomes. The report a support leader and a retention owner can act on joins the same failed conversations to later contacts and orders, grouped by contact driver:

The 150 failed contained conversations by contact driver, joined to later contacts and orders
Contact driverAbandonedCame back (how)Wrong answerWhat the order and billing data showsNext step
Refund status3024 (17 by phone)611 refunds not issued when the bot said they wereFix the refund-status lookup; update the refund-timeline article
Cancel or pause2214 (9 by phone)08 customers cancelled within 30 daysHand cancel requests to a person straight away
Promo codes1810 (by chat)95 checkouts abandonedPublish the current promo terms
Delivery status107 (4 by phone)03 orders still lateSend carrier data to Ops
Total8055 (30 by phone)15

Read across the rows and two different problems show up. Refund status is the bot’s biggest failure, and most of its repeat contacts come back as phone calls. Cancel requests are the retention problem: 8 of the 36 customers who abandoned or came back about cancelling left within 30 days, which no containment chart would show.

The same joined rows answer a different question for each team. Support sees which contact drivers to take away from the bot. The help-center owner sees which articles produced wrong answers. The retention owner gets the customers who asked to cancel while they can still be reached. Leadership sees what the 40% containment rate cost in refunds still owed and customers lost, next to the agent time it saved. That trade-off never appears on a containment chart.

What makes a chatbot handoff good, and how do you catch wrong answers?

Escalation isn’t necessarily a failure; the question is whether the handoff was appropriate and effective: did the bot escalate at the right point, collect useful information first and pass that context on, so the customer didn’t repeat themselves and the human resolved the issue? A bot that escalates 30% of conversations cleanly may serve customers better than one that “contains” 90% by refusing to give up.

Wrong answers are dangerous because they look like successful containment: the customer leaves and the dashboard records a win. Evaluate what the bot said against the relevant source of truth, such as current policy, help-center documentation or product behavior, and classify each answer as correct, incomplete, unsupported, or wrong. When an answer is wrong, check the article’s last-updated date and your release notes before blaming the bot: an article that still describes last month’s refund policy will keep producing the same wrong answer.

How does Rippit measure AI-agent outcomes?

Rippit reads every bot, chat, email and call you connect alongside your order, refund and billing data, keeps what it finds as structured, queryable data, and lets teams set up AI agents on it.

Define an analysis in plain language: Review every chatbot conversation. Determine whether the customer’s underlying issue was resolved, whether they abandoned first, whether the bot gave an incorrect answer, and whether any handoff preserved the context they already provided. Then add: Join each conversation to the customer’s orders and plan, and flag unresolved chats followed by a refund request or a cancellation. Rippit applies those classifications across every connected conversation and keeps the results, so you can ask which contact drivers have the highest unresolved-abandonment rate, or which ones come before the most cancellations. (How it works.)

Connect your conversation tools (Zendesk, Intercom, Ada, Decagon, Sierra) with pre-built integrations, and bring order and billing data in through Snowflake, the Bulk API or a CSV. Salesforce and Snowflake have pre-built connectors, and other tools such as HubSpot connect through MCP. Your team can read the results from Claude, ChatGPT or Cursor through Rippit’s hosted MCP server: an admin adds one connector URL, there is nothing to deploy, and each user sees only what they can already see in Rippit.

Klaviyo’s public customer story describes running AI across 100% of bot conversations to surface silent failures and find where humans were succeeding where the bot wasn’t. The team fed those patterns back into the bot’s training and routed an immediate handoff for intents the bot consistently struggled with, putting speed to resolution ahead of containment.

Evaluating AI support agents? See how Rippit compares with Intercom Fin, Sierra and Decagon.

FAQ: chatbot containment rate

Is containment rate still useful?

Yes, as an operational metric. It tells you how often the bot prevents a human interaction, which matters for staffing and cost modeling. Report it alongside the five Resolution Ledger metrics, so you can tell whether the bot reduced human work because it solved the problem or because the customer stopped talking.

What’s a good chatbot resolution rate?

There isn’t one universal number worth optimizing against, because definitions and windows differ. A more useful benchmark is your own AI agent over time, segmented by contact driver, including where humans materially outperform the bot.

How is this different from LLM observability?

LLM observability monitors the model’s system: latency, tokens, errors, traces. Conversation monitoring measures what happened to the customer: was the answer correct, was the problem solved, did the customer come back? You need both. A perfectly healthy model endpoint can still give customers bad answers.

Can I compare two AI-agent vendors this way?

Yes. Vendors define containment differently, so their dashboard percentages don’t compare. Apply the same definitions to both sets of conversations and you’re measuring the outcome using one standard. For the wider method, see the complete guide to conversation analytics.

Can’t I just paste our bot transcripts into Claude or ChatGPT?

Yes, for reading a batch of transcripts and reasoning about what went wrong. But the outcome usually isn’t in the transcript: the phone call three days later and the refund that was never issued sit in your help desk and billing system, and a pasted sample can’t tell you how often that happens. Connect Claude or ChatGPT to Rippit over MCP and they can query outcomes already recorded on every conversation you connect, joined to orders and refunds.

What’s the bottom line on chatbot containment rate?

Containment answers “did a human get involved?” The question you care about is “did the customer’s problem get solved?” Keep containment for planning, but measure AI-agent quality by the outcome. A customer closing the window isn’t a resolution. Sometimes they just gave up.

The ledger tells you what happened to each issue. The repeat-contact part is the one to automate first: check every chat the bot marked “resolved” for the customer coming back, and see what actually happened next.

Built example · checks every morning by default

Bot Outcome Check

Runs the ledger’s repeat-contact check on every chat your bot marked “resolved.” Every morning it matches each one to the same customer’s next conversations on every channel within 72 hours, checks your knowledge base, and sends:

  • How many “resolved” chats came back, and where: a ticket, an in-app chat, an email or a message in a shared Slack channel
  • The topics that came back most, with links to both conversations
  • Whether your knowledge base has a good answer, with draft help articles for the missing or outdated ones, for the knowledge owner to review
Sends to
SlackEmail

What is Rippit?

Joined to orders and billing, each bot outcome tells support, the help-center owner and the retention owner what to do next.

Rippit is a Growth Data Platform: it keeps enriching your customer conversations and other sources into structured data, connects them to order, billing and product data and each team’s own sources, and lets anyone set up agents on it. No engineer is needed to start. Connect with pre-built integrations or through our MCP, and ask from Rippit, Claude, ChatGPT or Cursor.

Support & AI agent conversations (bot, chat)Orders & refundsReopened contacts (phone, email)and more

Build your first agent on your customer and business data.

Try Rippit free
or request demo

Keep reading

More from the Rippit Library.

View the library →