Key takeaways
- Containment is not resolution. It tells you a human was avoided, not that the customer’s problem went away.
- True resolution needs both parts: a correct, grounded answer (or a clean handoff) and no repeat contact about the same issue within the window.
- 72 hours is a useful starting point for chat and email. Pick a window per workflow, document it and keep it stable.
- Measure every AI-handled conversation, not a sample, and keep the conversations behind each number.
- Report three numbers side by side: your measured resolution, the vendor-reported rate and your human reps’ rate on the same intents.
Why isn’t containment rate the same as resolution?
Containment answers one narrow question: did the conversation end without a human taking over? That’s a useful cost metric, but it can’t tell you whether the customer was helped. A customer who got the right answer and a customer who got an incomplete answer, closed the chat and came back tomorrow both count as contained. Only one was resolved.
CSAT doesn’t close the gap. Most customers never leave a rating, and a customer who gives up can still look resolved (Rippit docs: monitor your AI agent). For the full case against containment as a success metric, see Why Chatbot Containment Rate Misleads, and What to Measure Instead.
What is a defensible definition of true resolution?
Write a two-part test down before you measure anything. A conversation counts as truly resolved only when both parts pass.
Either test alone creates bad incentives. Checking only the conversation rewards a confident answer that didn’t work. Checking only repeat contact penalizes a good answer because the customer later wrote in about something unrelated.
How long should the repeat-contact window be?
72 hours is a useful starting point for many chat and email workflows. It’s long enough to catch the “that didn’t work, I’m back” return and short enough to avoid sweeping in unrelated new issues.
The right window depends on how quickly the outcome shows up. A password reset can be confirmed within hours; a billing correction or an insurance claim may need days. Set one window per workflow, document it and keep it stable across reporting periods. If the definition changes every month, the trend line stops meaning anything.
How do you measure true resolution across every AI conversation?
Defining resolution is easy; applying it consistently across thousands of conversations is the hard part. Sampling a few percent hides exactly the failures you’re looking for. A scalable audit follows five steps:
Step 1: Define the window
One per workflow, documented.
Step 2: Assemble the conversation set
Every conversation the AI agent handled in the period, plus later conversations from the same customers.
Step 3: Judge each conversation
The customer’s intent, whether the answer was correct, grounded and complete, and whether a handoff happened and was appropriate.
Step 4: Match repeat contacts by intent
Only a return about the same issue breaks resolution.
Step 5: Compare
Against the vendor-reported rate and against human reps on the same intents.
Keep the conversations behind every number. Aggregate metrics tell you where to look; conversation-level evidence tells you what actually happened.
What should a resolution audit show?
Report resolution by intent, not as one blended number. An overall rate of 84% can hide very different results underneath (illustrative figures, not customer data):
| Intent | True resolution (illustrative) |
|---|---|
| Password reset | 94% |
| Cancellation request | 82% |
| Billing adjustment | 71% |
| Account verification | 63% |
Then classify the failures behind each number: an incorrect or unsupported answer, an incomplete answer, an unnecessary or failed handoff, abandonment, or a repeat contact about the same issue. The point is a worklist, not a grade. Low resolution on account verification because the AI agent lacks the policy it needs calls for a different fix than low resolution caused by hallucinated answers.
How do you compare an AI agent with human reps fairly?
Match on intent. AI agents usually get the high-volume, repetitive intents; humans get the ambiguous and escalated ones, so a blended comparison flatters the bot. Take the intents both handle and, for each one, compare AI and human true-resolution rates and repeat-contact rates.
The goal isn’t to prove one is better. It’s to see where the AI agent is ready to carry more load and where it still needs work. For scoring the conversations themselves, see How Do You Evaluate Your AI Support Agent’s Conversations?
Why compare your number with the vendor’s?
The vendor’s resolution rate is a product metric, computed inside the system you’re evaluating, often from the bot’s own judgment that it finished. Definitions legitimately differ, so a gap isn’t proof the vendor is wrong; it’s something to investigate. The measurement is most credible from a layer with no stake in the result. Rippit doesn’t sell a customer-facing AI agent, so it reads your AI agent’s conversations independently of whichever vendor built it.
How does Rippit run a true-resolution audit?
Rippit lets anyone build AI agents on all of their conversation data, with no engineer required. Connect your customer conversation hubs in one click, including Zendesk, Intercom, Gong and more, then describe the audit in plain language:
Once it’s tested, save it as an Agent App that reruns on a schedule and delivers the results to Slack or email. A recurring run catches drift after a knowledge-base edit, a policy change or a new intent, instead of the week before a renewal. The same approach applies to human reps and to the wider AI agent monitoring playbook.
Frequently asked questions
Is containment rate useful at all?
Yes, as a cost metric: it tells you how often the AI agent avoided human handling. Just don’t present it as proof that the customer’s issue was resolved.
Why 72 hours?
It’s a practical starting point for fast-cycle chat and email, not a rule. Choose a window that matches how quickly the outcome can be observed, and say which window you used every time you report the number.
What if the customer comes back about a different issue?
That doesn’t count against resolution. Repeat contact is matched to the original intent, so only a return about the same issue breaks it.
Do I need to review every AI conversation by hand?
No. Apply the rubric automatically across every conversation, then have people inspect the evidence behind important findings, edge cases and failures.
Can I use the same definition for humans and AI agents?
Yes, when both populations are matched on the same intents. You still need to account for routing differences and case difficulty.
.webp)
