How to Audit AI Chatbot Escalations: A 5-Step Playbook and QA Rubric
Classify every escalation by root cause, then score the handoff itself, so you can separate avoidable bot failures from correct escalations and know what to fix, starting with the escalations on your highest-value accounts.
To audit AI chatbot escalations, analyze every escalated conversation and answer two separate questions: why did the bot escalate, and how well did it hand the customer off? Then join each escalation to the account’s ARR, SLA and renewal date, and flag the ones that land on your highest-value accounts. With Rippit, an agent you set up reviews AI agent and chatbot conversations alongside human ones, applies the root-cause codes and handoff rubric below to every conversation you connect, and keeps each result on the conversation, next to the account it came from.
Picture an enterprise admin who asks your bot the same SSO question three times, gets a wrong answer, and then emails their CSM a month before renewal. Your escalation rate counts that conversation once, exactly like a free-plan user who typed “agent” on the first message.
Key takeaways
- An escalation audit answers a different question than a containment dashboard: not how often the bot handed off, but why it handed off and whether it should have.
- Classifying every escalation into five root causes (retrieval failure, incorrect or stale knowledge, knowledge gap, capability or policy limit, and correct handoff or customer exit) turns one number into five fix lists with different owners.
- Handoff quality is measured separately: a correct escalation can still be a bad customer experience if context is lost or the customer has to repeat themselves.
- An escalation is clearly avoidable when the bot already had the knowledge it needed and failed to use it.
- Count the escalations your bot never reports: a chat marked “resolved” that comes back on another channel days later is a failed conversation too.
- Join each escalation to the account’s ARR, SLA and renewal date. The fix list stays ranked by cause; the avoidable escalations on high-value accounts also go to the account owner that week.
- Sampling a small share of escalations will reliably find your most common failure and reliably miss the rare, expensive one.
Why isn’t escalation rate enough?
Most teams running an AI support agent monitor some version of containment rate or escalation rate. Those metrics tell you what happened. They don’t tell you why.
Consider three conversations:
- A customer immediately types, “I want to talk to a person.”
- The bot can’t find an existing refund policy and escalates.
- The bot confidently gives the wrong refund policy, the customer pushes back, and only then does it escalate.
Customer: “I want to talk to a person.”
Dashboard counts: Escalation
RC-05 · Correct handoffMay need no fix at all.
Bot can’t find the existing refund policy, then hands off.
Dashboard counts: Escalation
RC-01 · Retrieval failureAvoidable. The AI / retrieval owner fixes it.
Bot confidently quotes the wrong refund policy. Customer pushes back, then it escalates.
Dashboard counts: Escalation
EQ-5 = 0 · Quality incidentWrong information left on record. Treat it separately.
Combine them into one escalation rate and you’ve created a metric without a remediation plan.
A useful escalation audit therefore produces two separate outputs:
Escalation root cause
Why did the bot hand off?
Handoff quality
Once it handed off, how well did it do it?
Here’s how to build both.
Step 1: Define every escalation you want to audit
Audit every conversation where the bot handed off, a human took over, or the customer asked for a person or gave up, not a sample.
Include conversations where:
- The bot explicitly handed off to a human.
- A human joined after the bot attempted to resolve the issue.
- The customer explicitly requested a human.
- The customer abandoned after an unsuccessful bot exchange where escalation should reasonably have occurred.
- The bot marked the chat resolved, but the same customer came back about the same problem on another channel (a ticket, an email to their CSM, a call or a shared Slack channel) within a few days.
Then analyze the entire population. Sampling made sense when reviewing conversations was expensive. If someone had to manually read 20,000 transcripts, reviewing 2% was often the only practical option.
Now an AI agent can evaluate every connected conversation, so a small sample no longer needs to be your primary measurement system.
Use samples to validate the classifier. Use the full dataset to measure the operation.
That’s especially important for AI-agent monitoring because the highest-volume problem isn’t necessarily the highest-risk problem. Incorrect pricing, bad policy guidance, mishandled sensitive situations, or other serious failures may occur infrequently enough to disappear from a small sample.
At Brex, which now reads every support ticket instead of sampling, Medha Mayawala, Business Development, puts it this way: “When we stopped QA, it wasn’t, let’s automate the scoring. It was, let’s stop doing what we’re doing and rebuild the whole thing.”
With Rippit, teams can create an agent that evaluates the complete set of connected bot conversations using criteria written in plain language. Connect your conversation tools, help desk and CRM with pre-built integrations, including Zendesk, Intercom and Gong, and billing data through Snowflake, the Bulk API or a CSV, then tell the agent what to check across every conversation you connect. Through our MCP, the agent can also read data from the rest of your stack, such as your knowledge base in Guru or a warehouse like Snowflake. That means an audit can check what the bot said against your help center, your CRM record or your verified knowledge base, not just the transcript, and against each customer’s plan, SLA, renewal date and the release they were on.
The result is a measurement of the actual population rather than an extrapolation from a handful of transcripts. For the wider monitoring approach, see The Complete Playbook for AI Agent Monitoring.
Step 2: Classify every escalation by root cause
Give every escalation exactly one primary root cause from the five codes below, instead of counting it in a single “escalated” bucket.
| Code | Root cause | What it looks like | Primary owner |
|---|---|---|---|
| RC-01 | Retrieval failure | The correct knowledge existed, but the bot failed to find or use it | AI / retrieval owner |
| RC-02 | Incorrect or stale knowledge | The bot found information, but the source itself was wrong, incomplete, or outdated | Knowledge / content |
| RC-03 | Knowledge gap | The request was in scope, but no approved answer existed | Knowledge / content |
| RC-04 | Capability or policy limit | The bot understood the request but wasn’t allowed or able to complete the required action | Product / operations |
| RC-05 | Correct handoff or customer exit | The request required a human by design, or the customer explicitly requested one | CX / AI-agent owner |
Assign one primary root cause to each escalation. If several things went wrong, choose the earliest issue that, had it been fixed, would have changed the outcome.
The distribution is more useful than the overall escalation rate because each bucket creates a different action:
- A spike in RC-01 means you investigate retrieval.
- A spike in RC-02 means you fix source content. Check your release notes first: a spike that starts the day after a release usually means an article still describes the old version.
- RC-03 identifies missing knowledge.
- RC-04 tells you where additional integrations, permissions, or automation could expand containment.
- And RC-05 may not need fixing at all.
That last point matters. The goal should not be zero escalations. The goal should be zero bad escalations.
Step 3: Score the quality of the bot-to-human handoff
A necessary escalation can still be a terrible customer experience, so score handoff quality separately from root cause.
Here’s a five-dimension rubric. Score each dimension from 0–2 for a total possible score of 10.
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| EQ-1: Timeliness | Bot loops three or more times before escalating | One unnecessary attempt | Escalates at first clear signal it can’t resolve |
| EQ-2: Context transfer | Human starts cold | Partial context transferred | Human receives full issue summary and prior attempts |
| EQ-3: Customer repetition | Customer repeats entire problem | Customer repeats some information | Customer repeats nothing |
| EQ-4: Expectation setting | No explanation of what happens next | Handoff acknowledged but unclear | Clear next step or wait expectation |
| EQ-5: Outcome integrity | Bot leaves incorrect or invented information on record | Answer is incomplete or ambiguous | No incorrect information left behind |
The total score is useful, but EQ-5 deserves special treatment. If the bot invented a policy, quoted the wrong price, gave unsafe instructions, or otherwise left the customer with incorrect information before handing off, a decent context transfer shouldn’t hide that.
Treat an EQ-5 score of zero as a separate quality incident.
On accounts with a contracted SLA, add one check next to the rubric: did a human pick up within the response time that customer pays for? A handoff can score 10 and still breach an enterprise SLA. That breach belongs to the account owner as well as the queue.
Zendesk’s own documentation describes evaluating AI agents with scorecards, either manually or automatically, and monitoring escalations through a BotQA dashboard (Evaluating the performance of AI agents using Zendesk QA). Check whether your setup can apply a custom rubric like this one across bot conversations in channels outside that helpdesk, and whether the same rubric can run against human conversations for comparison. For a side-by-side view, see Rippit vs. Zendesk.
Step 4: Separate avoidable from correct escalations
An escalation is avoidable when the bot could reasonably have resolved the request using the knowledge, permissions, and capabilities it was intended to have at that time.
That answers the question executives actually care about: how much of this could we have prevented? RC-01 is the clearest avoidable category. Parts of RC-02 and RC-03 are preventable too, but the fix sits with the business: the bot can’t use knowledge nobody maintained or wrote.
So don’t stop at a single “avoidable” percentage. Report the underlying causes with it. Using the worked example below (4,000 bot conversations):
31%
of bot conversations escalated
7.8%
bot failed to use knowledge it already had (RC-01)
7.7%
knowledge was stale or missing (RC-02, RC-03)
15.5%
policy, capability or correct handoffs (RC-04, RC-05)
Now you have something you can act on. “Escalation is 31%” starts a debate. “7.8% of conversations escalated even though we already had the information needed to answer them” creates a backlog.
Then cut the same numbers by account. A few avoidable escalations on your largest accounts can matter more to retention than a high rate on free or self-serve plans, so report two lines: the avoidable rate overall, and the avoidable escalations on accounts above your ARR threshold, on a premium SLA or renewing in your planning window.
Step 5: Save audit results as structured data
Save each audit result as a structured field on the conversation, so the next question doesn’t require a new analysis.
This is where escalation auditing becomes much more powerful than a one-time AI prompt. For every bot conversation, you can create structured fields such as:
Audit resultWhat happened in the conversationAccount and product data
Now those results aren’t trapped inside an AI-generated summary. They become part of the conversation dataset. That means you can ask:
“Which knowledge articles generated the most avoidable escalations this month?”
“Are customers with poor handoff scores more likely to contact us again within seven days?”
“Which AI-agent intents have the highest rate of RC-01 retrieval failures?”
“Did our RC-02 rate fall after last week’s help-center update?”
“Which avoidable escalations hit accounts up for renewal this quarter?”
“Which premium-SLA accounts waited longer than their contracted response time after a handoff this week?”
You don’t have to reconstruct the original analysis every time. The output of the AI becomes reusable data.
With Rippit, a non-technical user can define these fields and agents in natural language, run them across connected conversations, and schedule the analysis to update as conversations sync, with results delivered to email or Slack. Each knowledge gap reaches the team that owns it with the conversations behind it, so whoever files the issue in Linear, or Jira through a custom MCP connection, starts from the evidence. Ready-made agents for chatbot analysis, such as Bot Outcome Check, are in Rippit's use case library. For the broader method, see The Complete Guide to Conversation Analytics.
What counts as an avoidable, correct, late or missed escalation?
Steps 2 to 4 give every escalation a cause and a score. To report on them, and to send each one to the right owner, agree on the outcome labels too. The one most teams don’t count is the missed escalation. Only 14% of customer service issues are fully resolved in self-service, and in 43% of cases customers couldn’t find content relevant to their issue (Gartner, survey of 5,728 customers, August 2024). Many of the customers who don’t self-serve successfully never ask the bot for a person. They leave the chat, the bot records it as resolved, and they write to support or their CSM instead.
| Outcome | What it means | What confirms it | Who owns the fix |
|---|---|---|---|
| Avoidable | The bot should have resolved it (RC-01, and the parts of RC-02 and RC-03 where the answer should have existed) | An approved article or policy answered the question when the chat happened. Check its last-updated date against the chat. | AI / retrieval owner for RC-01; knowledge / content for RC-02 and RC-03 |
| Correct | The request needed a human by design, or the customer asked for one (RC-05) | The request type is on your list of human-only requests, or the customer asked for a person | No bot fix. The CX / AI-agent owner keeps the handoff score up. |
| Late | The right decision, made too late: EQ-1 scored 0 or 1 | The number of bot turns before the handoff, and the first message where the customer asked for a person or repeated themselves | AI-agent owner, through the escalation triggers |
| Missed | The bot marked the chat resolved, but the customer came back about the same problem on another channel within a set window, such as 72 hours | A later ticket, email, call or Slack message from the same customer (matched by email, account or workspace) on the same topic. A new problem doesn’t count. | AI-agent owner, plus knowledge / content when the article is missing |
| Quality incident | The bot left wrong or invented information on record: EQ-5 scored 0 | What the bot said contradicts the policy, price or article in force at the time, or the release notes for the version the customer is on | AI-agent owner, with the owner of the content it got wrong. Review every one; don’t sample. |
| High-value flag | Any of the above, on an account you can least afford to lose | ARR above your threshold, a premium plan or SLA, or a renewal date in your planning window, from the CRM and billing data | The owner above fixes the cause; the CSM or account owner hears about it the same week |
Late and missed escalations aren’t root causes, which is why they need labels of their own. A bot can pick the right cause and still be late, and a missed escalation never reaches Step 2 unless Step 1 includes it. The high-value flag isn’t an outcome either: it sits on top of one, so the fix list stays ranked by cause while the account team sees the accounts behind it. For how to match a returning customer across channels, phone included, see how to measure true resolution.
What does a completed escalation audit look like?
A completed audit turns one escalation rate into a ranked list of causes, each with an owner and a fix.
The example below is a fictional composite with round numbers for illustration; these are not measured results from any customer. Consider a fictional support team with 4,000 bot conversations in one week. Of those, 1,240 escalated, for an overall escalation rate of 31%. After classifying the conversations:
| Root cause | Escalations | Share |
|---|---|---|
| RC-01 Retrieval failure | 310 | 25% |
| RC-02 Incorrect/stale knowledge | 124 | 10% |
| RC-03 Knowledge gap | 185 | 15% |
| RC-04 Capability/policy limit | 385 | 31% |
| RC-05 Correct handoff/customer exit | 236 | 19% |
Now the 31% escalation rate tells a story. The team discovers that retrieval failures account for 310 conversations. More importantly, six knowledge articles are associated with most of those failures. Joining the results to account data shows those 310 failures came from 120 accounts, and 18 of them renew this quarter, so the six articles go to the top of the fix list.
Separately, handoff scoring shows that context transfer and customer repetition are the weakest dimensions. Those are two different problems with two different fixes: one team improves retrieval; another fixes the handoff payload so human agents receive the bot’s context.
That’s what an escalation audit should produce: not another dashboard, but a prioritized list of things to fix.
What the weekly audit should hand the account team
The fix list goes to the owners from Step 2. The same week’s audit, cut to high-value accounts, goes to customer success. Three accounts renewing this quarter had a failed bot conversation this week:
| Account | What happened | Outcome | Next step |
|---|---|---|---|
| Enterprise · $310K ARR · renews in 41 days | An admin asked the bot three times how to rotate an SSO certificate. The article existed. The bot escalated after the third loop. | Avoidable (RC‑01) and late (EQ‑1 = 0) | CSM checks in this week; the SSO article goes to the retrieval owner |
| Mid-market · $95K ARR · premium SLA | After last week’s release, the bot quoted the old API rate limit. The customer pushed back, then it escalated. | Quality incident (EQ‑5 = 0) | Correct the answer in writing; update the article; tag the CSM |
| Enterprise · $180K ARR · renews in 67 days | The bot marked an invoice question resolved. Two days later the finance admin emailed their CSM about the same invoice. | Missed | Billing fixes the invoice; the invoice article goes on the gap list |
Every row links to the conversations and the account record. The cause still goes to its fix owner; the account goes to its CSM before the renewal call, not after it. The highlighted account comes first because it combines the most ARR, the nearest renewal and two failures in one conversation.
The same fields serve every team that owns part of the fix, each with its own cut. The AI-agent owner filters RC-01 by intent to find what retrieval misses. The knowledge team sorts RC-02 and RC-03 by article and checks each spike against the release before it. Product reads RC-04 as a list of things customers expected the bot to do. CS filters the flags to accounts renewing this quarter and reads them next to the other churn risk signals in tickets and calls. Nobody reruns the analysis to get there.
Who should use Rippit to audit chatbot escalations?
Rippit analyzes every conversation you connect, so every number traces back to the conversations behind it. For a customer example, see the Klaviyo customer story.
At Klaviyo, Rick Franke, VP of Global Support Services, describes the change: “The product team was running at a five-minute mile, and we were running at a seven-minute mile. What Rippit has done is sped up the process so now we’re all running a four-minute mile.”
- Teams running an AI support agent at enough volume that manual escalation review has stopped being feasible
- CX and support ops leaders who need to show which escalations were avoidable
- CX and CS leaders who need to know which bot failures landed on high-value accounts before renewal
- Regulated teams that can’t rely on sampled review
- Teams whose bot handles a handful of conversations a day, where reading all of them by hand is faster
- Teams that want to instrument model-level traces and latency, which is an LLM observability problem, not a conversation analysis one
- Teams looking for a tool that rewrites bot flows for them: this audit tells you what to change, it doesn’t change it
If you run a vendor AI agent, see how Rippit compares with Intercom Fin, Sierra and Decagon.
FAQ: auditing AI chatbot escalations
How often should you audit AI chatbot escalations?
For a newly deployed or frequently changing AI agent, run the analysis at least weekly. Once performance stabilizes, the cadence can move to monthly, but severe quality conditions such as incorrect information should be monitored continuously or as frequently as your operation requires. Apply the same definitions consistently over time so you can see whether the underlying failure modes are actually improving. An agent built in Rippit can run the audit on a schedule and deliver results to email or Slack.
Is escalation rate a bad metric?
It is an incomplete one. Escalation rate mixes correct handoffs with failures, so it can fall while customer experience gets worse, for example if the bot stops escalating cases it should escalate. Track avoidable escalation rate and handoff quality alongside it. If the rate falls while more “resolved” chats come back on other channels, the bot is hiding escalations, not preventing them.
How do you flag escalations on high-value accounts?
Join each escalation to the account record: ARR, plan and SLA tier from billing, and the renewal date from the CRM. Set a threshold, such as ARR above a set amount, a premium SLA or a renewal in the next two quarters, and flag any avoidable, late, missed or incident outcome on those accounts. Keep the fix list ranked by root cause, and send the flagged accounts to their CSM or account owner the same week.
How is this different from LLM observability?
LLM observability and conversation analysis answer different questions. Observability tools inspect the system: model calls, traces, latency, token usage, tool calls and errors. An escalation audit inspects the customer interaction: did the customer get the right answer, why couldn’t the bot resolve the request, was escalation appropriate, was context preserved, did the customer have to repeat themselves, and what ultimately happened. You may need both. A technically successful model call can still produce a terrible customer experience.
Can you audit chatbot escalations without an engineer?
Yes, to start. The root-cause taxonomy and QA rubric above are written as plain-language judgments. In Rippit, teams can turn judgments like these into agents and structured fields without writing SQL or building their own data pipeline. Rippit ingests the underlying conversation data, runs the analysis across every conversation you connect, and maintains the results as part of the conversation dataset. Rippit hosts its MCP server itself, so there is nothing to deploy (MCP server overview). That makes the audit repeatable instead of a one-time project.
Can I run this audit with Claude or ChatGPT?
Yes, for the reasoning. Give Claude or ChatGPT a set of escalations and this rubric, and it will classify and score them well; code execution will also count any column you export exactly. The limit is the population. Once your escalations don’t fit in its context, the assistant retrieves the most relevant ones or works from summaries, so the share of RC-01 failures depends on which conversations it read, and its calls aren’t saved as fields on each conversation that stay current as new ones arrive. Connected to Rippit over MCP, the assistant queries the root cause, rubric scores and account fields Rippit already keeps on every escalation you connect. For a fuller comparison, see Rippit vs. Claude.
What if our chatbot isn’t in Intercom or Zendesk?
Run the playbook anyway: the taxonomy and rubric are vendor-neutral and work against any transcript your team can export. Rippit connects your conversation tools with pre-built integrations, including Zendesk, Intercom and Gong, and through our MCP the agent can also read data from the rest of your stack, such as Salesforce and Snowflake, or HubSpot through MCP, plus your own data. New integrations ship regularly; see the current list on the integrations page and in the supported integrations docs.
The bottom line
Don’t optimize your AI agent for the lowest possible escalation rate. Optimize for correct resolution when the bot should resolve, and a clean handoff when it shouldn’t.
- Measure why every escalation happened.
- Measure the quality of the handoff separately.
- Turn those measurements into persistent data.
- Flag the failures on high-value accounts, and tell the account owner that week.
- Then fix the failure modes that actually matter.
That’s how escalation rate becomes an operating system instead of a vanity metric.
An audit you run once a quarter tells you what broke last quarter. To apply the same codes and rubric to every escalation as it happens, set the audit up as an agent. For the “resolved” chats that come back on another channel, add the daily repeat-contact check from the containment guide.
A custom escalation audit agent
Run this audit on every escalation, every day. Each day it applies your root-cause codes and the five-part handoff rubric to every bot conversation you connect, checks what the bot said against your knowledge base and policies, and sends:
- Escalations by root cause, with the avoidable share and its trend
- Handoff scores by dimension, with every EQ-5 = 0 conversation listed as a quality incident
- The knowledge articles behind the most RC-01 and RC-02 escalations, for the knowledge owner to fix
- Flagged escalations on key accounts, with links to the conversations, for the CSM
What is Rippit?
Every escalation this audit classifies is data more than one team needs: a fix for the AI-agent owner, an article for the knowledge team and, on the accounts that matter most, a call for the CSM.
Rippit is a Growth Data Platform: it keeps enriching your customer conversations and other sources into structured data, connects them to CRM, billing and product data and each team’s own sources, and lets anyone set up agents on it, without needing an engineer to start. Connect with pre-built integrations or through our MCP, and ask from Rippit, Claude, ChatGPT or Cursor.
Build your first agent on your customer and business data.
Sources
- Zendesk Help Center · Evaluating the performance of AI agents using Zendesk QA
- Rippit · Klaviyo customer story
- Rippit · Integrations
- Rippit Docs · Supported integrations
- Rippit Docs · MCP server overview
- Rippit · Use case library
- Gartner, survey of 5,728 customers (August 2024)
Keep reading
More from the Rippit Library.
Conversation Analytics, From Reading Every Call to Acting on It
What it is, how AI reads every conversation, and what changes when conversations sit next to CRM, product and market data.
Measure What Your AI Agent Actually Resolves
Find the bot conversations that weren’t truly resolved, including customers who came back on another channel, and see which accounts need attention.
Catch the Wrong Answers Your AI Agent Gives Customers
Five checkable hallucination types, the signals that find them and the source behind each, with the accounts, revenue and compliance exposure they touch.