---
title: "Why Chatbot Containment Rate Misleads, and What to Measure Instead"
description: "Containment counts abandoned chats as wins. Five metrics that measure confirmed resolution, repeat contacts and handoff quality instead."
canonical: https://www.rippit.com/guides/chatbot-containment-rate-misleading-what-to-measure-instead
type: Guide
datePublished: 2026-10-02
---

# Why Chatbot Containment Rate Misleads, and What to Measure Instead

Containment tells you whether a chatbot conversation ended without a human. It does not tell you whether the customer’s problem was solved, or whether they called back, asked for a refund or cancelled.

By Mike Nucci · Published October 2, 2026

Containment rate tells you whether a chatbot conversation ended without a human. It does **not** tell you whether the customer’s problem was solved. A customer whose question was answered correctly can count as contained. So can a customer who received a wrong answer, got frustrated, closed the chat, and emailed support the next morning. Only one was a success.

**Containment measures the absence of a human. It does not measure the presence of an answer.** Measure the customer outcome instead: confirmed resolution, unresolved abandonment, repeat contact, handoff quality, and wrong answers. Then join each outcome to the customer’s orders, refunds and plan, so you can see which bot failures end in a refund request or a cancellation.

With Rippit, an agent you set up reviews every bot, chat and human conversation you connect, so you can see where the bot resolves, escalates or gets answers wrong.

Picture a subscriber who asks your bot how to cancel an add-on. The bot explains, the chat ends, and the dashboard counts a contained conversation. Three days later the same customer calls, still being billed, and asks for a refund. The bot logged a success and the phone team logged a resolved call. Only the billing record shows what actually happened.

## Key takeaways

- Containment measures the absence of a human, not the presence of an answer, so an abandoned chat and a solved problem score identically.
- Replace one number with five: confirmed resolution, unresolved abandonment, repeat contact rate, handoff quality and wrong-answer rate.
- Confirmed resolution needs a window and a repeat-contact check; without both, you are measuring session endings.
- Match repeat contacts on every channel, including phone. A “contained” chat followed by a call about the same order is a failed conversation.
- Join outcomes to orders, refunds and cancellations to see which bot failures cost you customers, not just agent time.

## What does chatbot containment rate actually measure?

Containment asks one question: did this interaction end without escalating to a human agent?

The problem comes when containment becomes a proxy for **resolution**.

Consider three conversations:

- **Conversation A:** The bot gives the correct answer and the customer leaves.
- **Conversation B:** The bot confidently gives the wrong answer and the customer accepts it.
- **Conversation C:** The bot repeats the same troubleshooting steps until the customer stops responding.

All three can appear contained. Good [AI-agent monitoring](https://www.rippit.com/playbooks/ai-agent-monitoring) asks what actually happened to the customer’s issue.

**BOT DASHBOARD**

Conversation A

Contained

Conversation B

Contained

Conversation C

Contained

Containment: 3 of 3

**CONVERSATION EVIDENCE · SAME 3 CHATS**

A · Chat #31842

Confirmed resolution

Correct answer, customer confirmed it worked, no repeat contact.

B · Chat #31907

Wrong answer

Bot’s answer conflicts with current policy. The customer accepted it.

C · Chat #32015

Unresolved abandonment

“I already tried that.” Same steps repeated, then the customer stopped responding.

*All three conversations count as contained; only one solved the customer’s problem.*

## What should you measure instead of chatbot containment? The Resolution Ledger

No single metric replaces it. You need five that describe different failure modes.

**The Resolution Ledger: five metrics to report alongside containment**

| Metric | What it measures | Measurement window | Failure signal |
| --- | --- | --- | --- |
| Confirmed resolution | Evidence that the customer’s issue was resolved, with no same-issue repeat contact | Session + the repeat-contact window | Issue appears unresolved or customer returns with the same problem, in any channel |
| Unresolved abandonment | Customer stops engaging before there is evidence of resolution or successful handoff | Session | Conversation ends without evidence the issue was solved |
| Repeat contact rate | Customer contacts support again about the same underlying issue | Often 72 hours; adapt to your business | Same customer, same issue, another conversation in any channel, including a phone call |
| Handoff quality | Whether escalation happened at the right time and context carried to the human | Handoff | Customer has to repeat information already given to the bot |
| Wrong-answer rate | Bot gives information that conflicts with the applicable policy, documentation, or known facts | Session | Materially incorrect or unsupported answer |

Set the window by product (a weekly-use B2B tool needs longer than checkout). The important part is having a window at all. For the full method, including repeat contacts that come back by phone, see [how to measure true resolution for AI agent conversations](https://www.rippit.com/guides/measure-true-resolution-ai-agent-conversations).

> A resolution metric without a repeat-contact window is just a session-end counter with a better name.

Customers don’t always come back, so confirmed resolution should combine signals: the requested action was completed, the customer confirmed the fix, and no same-issue contact followed during the window. “The requested action was completed” is something you can check: the refund shows as issued, the order was changed, the add-on was removed.

## Why measure the customer’s issue, not just the chatbot session?

Containment is session-centric. Customers are not.

A customer chats with the bot about a failed payment on Monday, chats again on Tuesday, and emails support on Wednesday, where a human finally fixes it. A dashboard sees three interactions; the customer had **one problem that took three conversations to resolve**.

The better data model is **conversation → issue → outcome → repeat contact**, rather than **session → contained / not contained**. Because customers rephrase (“My payment won’t go through,” then “Still can’t complete checkout”), matching ticket tags undercounts; analyze [each conversation’s contact driver](https://www.rippit.com/guides/why-customers-contact-support-at-scale) to decide whether a later one is the same problem.

At Resident, Michelle Zimmerman, Senior Quality Data Analyst, puts the stakes this way: “Understanding where the journey breaks down impacts retention, future sales, referrals, everything that matters. Rippit finally gives us that visibility.”

## What should you join each chatbot outcome to?

An outcome label tells you what happened to the issue. Joined to the customer’s later contacts, orders and plan, it tells you what the failure cost and who should fix it.

**Five joins that turn a chatbot outcome into a retention signal**

| Join | What it adds | Where it comes from | The question it answers |
| --- | --- | --- | --- |
| Later contacts, every channel | The same customer’s next ticket, email or phone call about the same issue, matched by email, account or order number within your window | Help desk, email, phone tickets and call records | Which “contained” chats came back, and on which channel? |
| Orders and refunds | The order, refund or charge the chat was about, and whether it was completed | Order and billing systems | Did the customer get what the bot said they would? |
| Plan and cancellations | The customer’s plan and tenure, and whether they cancelled or downgraded after the chat | Billing and subscription data | Which bot failures come before a cancellation? |
| Help-center docs and release notes | The article or policy the bot answered from, its last-updated date, and the release the customer was on | Knowledge base, release notes | Was a wrong answer the bot’s fault or the article’s? |
| Handoff record | What the bot passed to the human, and how long the customer waited | Help desk handoff fields | Did the human start cold? |

Start with later contacts on every channel, including phone. A chat-only dashboard can’t see the customer who gave up on the bot and called instead.

## What can a 40% containment rate look like after you measure outcomes?

Consider a hypothetical example.

An AI support agent handles **1,000 conversations** in a month, and its dashboard reports **40% containment**: 400 conversations ended without a human. Inspect those 400 and give each one a single primary outcome, so the four counts add up to 400. Failure signals overlap in practice (a wrong answer is often followed by a repeat contact), so pick one rule for which outcome wins and apply it every time:

- 80 ended in unresolved abandonment.
- 55 were followed by another contact about the same issue.
- 15 contained a materially incorrect answer.
- 250 showed evidence of resolution and survived the repeat-contact check.

The headline metric was **containment: 40%** of all conversations. The more meaningful one was **confirmed resolution: 25%** of all conversations (250 of 1,000), which is 62.5% of the contained ones. A containment rate tells you where the conversation ended. An outcome analysis tells you **what happened to the problem**.

**DASHBOARD HEADLINE**

**40%**

Containment · 400 ended without a human

**OUTCOME ANALYSIS**

**25%**

Confirmed resolution · 250 of 1,000

**Inside the 400 contained conversations (one primary outcome each)**

| Outcome | Conversations | Share of contained |
| --- | --- | --- |
| Confirmed resolution | 250 | 62.5% |
| Unresolved abandonment | 80 | 20% |
| Repeat contact, same issue | 55 | 13.75% |
| Materially incorrect answer | 15 | 3.75% |

*A 40% containment rate becomes 25% confirmed resolution once each contained conversation is checked for its outcome.*

### What the weekly outcome report should show

The ledger counts outcomes. The report a support leader and a retention owner can act on joins the same failed conversations to later contacts and orders, grouped by contact driver:

**The 150 failed contained conversations by contact driver, joined to later contacts and orders**

| Contact driver | Abandoned | Came back (how) | Wrong answer | What the order and billing data shows | Next step |
| --- | --- | --- | --- | --- | --- |
| Refund status | 30 | 24 (17 by phone) | 6 | 11 refunds not issued when the bot said they were | Fix the refund-status lookup; update the refund-timeline article |
| Cancel or pause | 22 | 14 (9 by phone) | 0 | 8 customers cancelled within 30 days | Hand cancel requests to a person straight away |
| Promo codes | 18 | 10 (by chat) | 9 | 5 checkouts abandoned | Publish the current promo terms |
| Delivery status | 10 | 7 (4 by phone) | 0 | 3 orders still late | Send carrier data to Ops |
| **Total** | **80** | **55 (30 by phone)** | **15** |  |  |

Read across the rows and two different problems show up. Refund status is the bot’s biggest failure, and most of its repeat contacts come back as phone calls. Cancel requests are the retention problem: 8 of the 36 customers who abandoned or came back about cancelling left within 30 days, which no containment chart would show.

The same joined rows answer a different question for each team. Support sees which contact drivers to take away from the bot. The help-center owner sees which articles produced wrong answers. The retention owner gets the customers who asked to cancel while they can still be reached. Leadership sees what the 40% containment rate cost in refunds still owed and customers lost, next to the agent time it saved. That trade-off never appears on a containment chart.

## What makes a chatbot handoff good, and how do you catch wrong answers?

**Escalation isn’t necessarily a failure; the question is [whether the handoff was appropriate and effective](https://www.rippit.com/playbooks/audit-ai-chatbot-escalations)**: did the bot escalate at the right point, collect useful information first and pass that context on, so the customer didn’t repeat themselves and the human resolved the issue? A bot that escalates 30% of conversations cleanly may serve customers better than one that “contains” 90% by refusing to give up.

Wrong answers are dangerous because they look like successful containment: the customer leaves and the dashboard records a win. [Evaluate what the bot said](https://www.rippit.com/guides/evaluate-ai-support-agent-conversations) against the relevant source of truth, such as current policy, help-center documentation or product behavior, and classify each answer as correct, incomplete, unsupported, or wrong. When an answer is wrong, check the article’s last-updated date and your release notes before blaming the bot: an article that still describes last month’s refund policy will keep producing the same wrong answer.

## How does Rippit measure AI-agent outcomes?

Rippit reads every bot, chat, email and call you connect alongside your order, refund and billing data, keeps what it finds as structured, queryable data, and lets teams set up AI agents on it.

Define an analysis in plain language: *Review every chatbot conversation. Determine whether the customer’s underlying issue was resolved, whether they abandoned first, whether the bot gave an incorrect answer, and whether any handoff preserved the context they already provided.* Then add: *Join each conversation to the customer’s orders and plan, and flag unresolved chats followed by a refund request or a cancellation.* Rippit applies those classifications across every connected conversation and keeps the results, so you can ask which contact drivers have the highest unresolved-abandonment rate, or which ones come before the most cancellations. ([How it works](https://www.rippit.com/how-it-works).)

Connect your conversation tools (Zendesk, Intercom, Ada, Decagon, Sierra) with pre-built integrations, and bring order and billing data in through Snowflake, the Bulk API or a CSV. Salesforce and Snowflake have pre-built connectors, and other tools such as HubSpot connect through [MCP](https://docs.rippit.com/mcp-server/overview). Your team can read the results from Claude, ChatGPT or Cursor through Rippit’s [hosted MCP server](https://docs.rippit.com/mcp-server/overview): an admin adds one connector URL, there is nothing to deploy, and each user sees only what they can already see in Rippit.

[Klaviyo’s public customer story](https://www.rippit.com/customer-story/klaviyo) describes running AI across [100% of bot conversations](https://www.rippit.com/guides/qa-100-percent-of-support-conversations) to surface silent failures and find where humans were succeeding where the bot wasn’t. The team fed those patterns back into the bot’s training and routed an immediate handoff for intents the bot consistently struggled with, putting speed to resolution ahead of containment.

Evaluating AI support agents? See how Rippit compares with [Intercom Fin](https://www.rippit.com/comparisons/rippit-vs-intercom-fin), [Sierra](https://www.rippit.com/comparisons/rippit-vs-sierra) and [Decagon](https://www.rippit.com/comparisons/rippit-vs-decagon).

## FAQ: chatbot containment rate

### Is containment rate still useful?

Yes, as an **operational metric**. It tells you how often the bot prevents a human interaction, which matters for staffing and cost modeling. Report it alongside the five Resolution Ledger metrics, so you can tell whether the bot reduced human work **because it solved the problem** or because the customer stopped talking.

### What’s a good chatbot resolution rate?

There isn’t one universal number worth optimizing against, because definitions and windows differ. A more useful benchmark is your own AI agent over time, segmented by contact driver, including where humans materially outperform the bot.

### How is this different from LLM observability?

LLM observability monitors the model’s system: latency, tokens, errors, traces. Conversation monitoring measures what happened to the customer: was the answer correct, was the problem solved, did the customer come back? You need both. A perfectly healthy model endpoint can still give customers bad answers.

### Can I compare two AI-agent vendors this way?

Yes. Vendors define containment differently, so their dashboard percentages don’t compare. Apply the same definitions to both sets of conversations and you’re measuring the outcome using one standard. For the wider method, see [the complete guide to conversation analytics](https://www.rippit.com/playbooks/the-complete-guide-to-conversation-analytics).

### Can’t I just paste our bot transcripts into Claude or ChatGPT?

Yes, for reading a batch of transcripts and reasoning about what went wrong. But the outcome usually isn’t in the transcript: the phone call three days later and the refund that was never issued sit in your help desk and billing system, and a pasted sample can’t tell you how often that happens. Connect Claude or ChatGPT to Rippit over MCP and they can query outcomes already recorded on every conversation you connect, joined to orders and refunds.

## What’s the bottom line on chatbot containment rate?

**Containment answers “did a human get involved?” The question you care about is “did the customer’s problem get solved?”** Keep containment for planning, but measure AI-agent quality by the outcome. A customer closing the window isn’t a resolution. Sometimes they just gave up.

The ledger tells you what happened to each issue. The repeat-contact part is the one to automate first: check every chat the bot marked “resolved” for the customer coming back, and see what actually happened next.

**Built example · checks every morning by default**

**Bot Outcome Check**

Runs the ledger’s repeat-contact check on every chat your bot marked “resolved.” Every morning it matches each one to the same customer’s next conversations on every channel within 72 hours, checks your knowledge base, and sends:

- How many “resolved” chats came back, and where: a ticket, an in-app chat, an email or a message in a shared Slack channel
- The topics that came back most, with links to both conversations
- Whether your knowledge base has a good answer, with draft help articles for the missing or outdated ones, for the knowledge owner to review

Sends to: Slack · Email

## What is Rippit?

Joined to orders and billing, each bot outcome tells support, the help-center owner and the retention owner what to do next.

**Rippit is a Growth Data Platform: it keeps enriching your customer conversations and other sources into structured data, connects them to order, billing and product data and each team’s own sources, and lets anyone set up agents on it.** No engineer is needed to start. Connect with pre-built integrations or through our [MCP](https://docs.rippit.com/mcp-server/overview), and ask from Rippit, Claude, ChatGPT or Cursor.

Support & AI agent conversations (bot, chat) · Orders & refunds · Reopened contacts (phone, email) · and more

Build your first agent on your customer and business data.

[Try Rippit free](https://app.rippit.com/signup) [or request demo](https://www.rippit.com/request-demo)

## Keep reading

- [The Growth Data Your AI Agent Needs Doesn’t Exist Yet](https://www.rippit.com/guides/growth-data-ai-agent-needs-doesnt-exist-yet) (Next in Guides)
- [Measure What Your AI Agent Actually Resolves](https://www.rippit.com/guides/measure-true-resolution-ai-agent-conversations) (Guide)
- [Catch the Wrong Answers Your AI Agent Gives Customers](https://www.rippit.com/guides/detect-ai-chatbot-hallucinations-customer-conversations) (Guide)

Setup steps for AI agents: https://www.rippit.com/setup.md
