ANSWER

How to QA 100% of Support Conversations Instead of 2%

Stop using humans as the sampling mechanism: let AI score every eligible conversation against your existing rubric, then point your reviewers at calibration, exceptions and coaching.

Try Rippit free
or request demo
By Lauren Alexander · Published October 2, 2026

To QA 100% of support conversations, stop using humans as the sampling mechanism: let AI score every eligible conversation against your existing rubric, then point your reviewers at calibration, exceptions and coaching. With Rippit, a QA agent you set up scores every conversation you connect and ties each score to the account, so a critical failure on an account nearing renewal reaches the CSM as well as the team lead.

Every QA program starts roughly the same way. A reviewer grades 20 of 1,000 tickets, and the number goes onto a slide: Quality: 92%. The other 98% were never scored. Sampling was the only option when a person had to read each one. AI changes the constraint.

Key takeaways

  • A failure that happens 5 times in 10,000 conversations has roughly a 90% chance of never appearing in a 2% sample.
  • Automate the checkable criteria, let AI score interpretive ones under human calibration, and keep accountability human.
  • Reviewers move to calibration, exceptions, coaching and root cause.
  • Scored at 100%, QA becomes a dataset you can combine with contact drivers and resolution, and with each account’s plan, ARR and renewal date.

What does a 2% QA sample actually miss?

Often, every instance of a rare failure.

Suppose your team handles 10,000 conversations and randomly reviews 2%, or 200. (QA sampling math shows how these numbers are worked out.) Here’s the approximate probability that those 200 reviews contain zero examples of a failure at different rates:

Chance a 2% random sample (200 of 10,000 conversations) misses every instance of a failure
Failure rateInstances in 10,000Approx. chance a 200-conversation sample misses all of them
5%500<0.01%
1%100~13%
0.5%50~36%
0.2%20~67%
0.1%10~82%
0.05%5~90%

These are approximate probabilities under random sampling, not observed customer data: the chance of missing every instance is about 0.98 raised to the number of instances.

Rare events are often the failures you care most about: a skipped disclosure, a promise policy doesn’t allow, a misapplied refund policy, or an AI agent giving a wrong answer.

2% RANDOM SAMPLE
200 of 10,000 reviewed

The 5 failures sit outside the sample.

~90%

chance none of the 5 serious failures is ever reviewed

Serious failure, never reviewed
100% COVERAGE
10,000 of 10,000 scored

Every eligible conversation scored against the rubric.

5 of 5

serious failures scored, each with the conversation behind it

Serious failure, caught
The same 10,000 conversations with 5 rare failures: a 2% sample usually misses all of them; scoring every conversation surfaces each one.

Klaviyo, handling over 600,000 incidents a year, had been reviewing under 2% of them (Klaviyo story).

Use samples to calibrate the scorecard. Use the full dataset to measure quality.

What does “100% QA” actually mean?

It means every eligible conversation in the population you’ve defined and connected is evaluated. “100% of Zendesk support tickets from September 1–30” is a defensible measurement. “100% of customer conversations” is only accurate if every relevant source is actually represented.

What changes when you QA every support conversation?

QA becomes a dataset.

Every eligible conversation can carry structured fields: overall QA score, policy adherence, resolution, required disclosure completed, correct next step, escalation quality, customer effort, agent, contact driver and root cause, plus the account’s plan, ARR and renewal date from your CRM and billing data.

Instead of “Our quality score was 88% last month,” you can ask: Which contact drivers produced our lowest QA scores? Which reps need coaching specifically on escalation quality? Which accounts renewing this quarter had a conversation fail a critical check? Did one criterion start failing the day after a release?

Which QA criteria should AI score?

Not every scorecard line should be treated the same way.

Divide the rubric into three tiers:

Three tiers of QA criteria and who owns each score
TierCriteriaWho owns the score
Automate the checkableRequired disclosure, verification step, correct policy, correct next step, resolution confirmed, escalation path, prohibited languageAI, with the evidence
AI scores, humans calibrateProblem understood, clear explanation, customer effort, de-escalation, concern acknowledgedAI for measurement and prioritization; a human confirms before a score drives coaching or performance
Keep accountability humanFormal performance actions, disputed scores, novel situations, rubric calibration, ambiguous edge casesA person

Interpretive criteria need stronger calibration because reasonable reviewers disagree about what “good” looks like. AI can own measurement without owning accountability. (More in the complete guide to conversation analytics.)

How do you know whether AI QA scores are trustworthy?

Calibrate before rolling scores out.

Have experienced reviewers and the AI independently score a shared set of conversations with your existing rubric, then compare results criterion by criterion, not just overall.

When they disagree, inspect why. Sometimes the model is wrong; sometimes two humans disagree too, which says something about the rubric. “Demonstrated empathy” leaves enormous room for interpretation. “Did the rep acknowledge the customer’s frustration before the next troubleshooting step?” is easier to apply consistently. Calibration forces you to make the definition of quality explicit.

Where do human QA reviewers go when AI scores every conversation?

Nowhere. Their work gets more valuable:

  • Calibration: test whether AI and reviewers interpret the rubric consistently.
  • Exceptions: investigate the lowest-scoring, highest-risk or most unusual conversations.
  • Coaching: instead of “Your QA score is 84%,” show the two behaviors behind lower scores, with examples from the rep’s own conversations.
  • Root cause: if the same failure appears 400 times, the policy, macro or help article is probably wrong, not 400 reps. If failures start the day an article changed, fix the article before you coach anyone.

QA teams stop being a sampling function and start being a quality function.

Why does traceability matter for automated QA scores?

Every automated QA score should carry evidence.

If the system scores Required disclosure: FAIL, a reviewer should be able to open the conversation and see why. People won’t trust a score they can’t challenge, and evidence shows whether the model, the rubric or the source data was at fault.

TICKET #48217 · QA AGENT · SCORED AUTOMATICALLY
Flagged: critical policy requirement

STRUCTURED FIELDS

Required disclosure FAIL

Correct next step Pass

Escalation quality Needs review

Contact driver Refund request

EVIDENCE · REQUIRED DISCLOSURE

Customer
“Can you just refund it to a different card?”

Agent
“Done, it’s on its way.”

Why it failed: the required disclosure was never given before the refund was processed.

Open full conversationDispute score
One scored conversation: each criterion becomes a structured field, and a failing score links to the exact lines behind it so a reviewer can verify or dispute it.

What does a good weekly QA output look like?

One row per rep and one for the AI agent, each score backed by conversations, with the flags that touch your key accounts on top.

Here is one week for a B2B support team. The team score is 88, up 3 points. 63 conversations were flagged, 18 of them at key accounts, and those 18 go to the account’s CSM as well as the team lead.

Weekly QA rows, every conversation scored
RepConversationsKnowledge-base accuracyScoreCoaching note
Rep A9698%94 (↑3)Strength: gives a time for every update
Rep B8891%81 (↓4)Loop in the CSM when a key account is upset
Rep C10284%76 (↓2)Use the current SSO article, not the old one
AI agent2,48088%79 (↑1)Retire the export article it keeps linking

What makes these rows useful:

  • Knowledge-base accuracy sits next to the score. Rep C has the lowest accuracy and a note about an outdated SSO article: retire the old article first, then coach.
  • The AI agent is scored on the same rubric, so a bad article it keeps linking shows up in the same report as the reps.
  • Each coaching note is a pattern across the week, with the conversations attached, not the last bad ticket.
  • Flags carry the account, so Rep B’s note is about looping the CSM in.

How does Rippit handle QA?

Rippit treats QA as one type of analysis on the same customer and business data used for customer insights, churn signals and chatbot monitoring. An admin connects your help desk and call recordings with pre-built integrations, including Zendesk, Intercom and Gong, plus Salesforce through a pre-built connector or HubSpot through our MCP, and billing data through Snowflake, the Bulk API or a CSV. Then a team sets up a QA agent in plain language:

“Evaluate every support conversation against these eight criteria. For each criterion, return the score and the evidence supporting it. Flag conversations that fail any critical policy requirement and summarize the most common failure patterns each week.”

Add your rubric, help articles and policy docs by CSV or the Bulk API, so a “correct policy” score is checked against your own policies. The agent can also read data from the rest of your stack, such as Snowflake through a pre-built connector, or other tools through our MCP. See how it works, ready-made agents, and why Rippit reads 100% of conversations. Brex’s public story describes rebuilding its QA playbook from the ground up.

Comparing QA tools? See how Rippit stacks up against Zendesk QA, Level AI and Solidroad, or read our guide to Zendesk QA alternatives.

Which agents should you run alongside QA?

Scoring is one agent. Others build on the same conversation and account data, each for the person who acts on what it finds:

Agents on the same conversation and account data
AgentWho it’s forWhat it addsRuns
AutoQA · Built exampleQA managersEvery rep and the AI agent scored on your rubric, answers checked against the knowledge base and policies, flags with the reasonDaily (every morning)
Agent Coaching Brief · Built exampleTeam leadsFor each rep, the two habits to work on this week and a strength, with example conversationsWeekly (Mondays)
Bot Outcome Check · Built exampleAI agent ownersBot chats marked “resolved” that came back on another channel within 72 hours, by topicDaily (every morning)
A custom escalation-warning agent · Example you can buildCX leaders and CSMsTickets likely to escalate to an executive, from tone, repeat contacts, ARR and renewal dateDaily
Churn-Risk Bug Watch · Built exampleProduct and engineering leadsBugs hitting accounts that renew in the next 90 days, with the conversations behind eachWeekly

Start with AutoQA, then add the agent whose output someone will act on this week. For most QA teams, that’s the coaching brief.

FAQ: QA for 100% of support conversations

Does full-coverage QA mean every score is automatically correct?

No. Full coverage solves the coverage problem; calibration solves measurement; humans handle accountability. A bad rubric applied to 100% of conversations is still a bad rubric.

Can AI QA scores be used for performance reviews?

Use caution. Before an AI-derived criterion feeds a formal employment decision, validate it against experienced reviewers and set up a process for disputed scores.

Do you still need manual QA?

Yes, but much less random manual scoring. Humans stay on calibration, coaching, appeals and improving the rubric.

Is this different from what MaestroQA did?

MaestroQA is Rippit’s former name, not a different product; QA customers use the same platform. QA is now one of several agents teams build on the same data.

How long does it take to get the first scores?

Connect your help desk and other conversation tools with pre-built integrations, including Zendesk, Intercom and Gong, then describe the agent in plain language; it scores on a schedule. Budget more time for calibration than setup.

Can I score conversations with Claude or ChatGPT instead?

Yes, for the conversations they load or retrieve, and they’re good at applying a rubric and explaining a score. But scoring what an assistant retrieves is the 2% problem again: the conversations it didn’t load go unscored, and a score made in a chat isn’t saved as a field you can trend next month. Connect Claude or ChatGPT to Rippit over MCP and they can query scores already recorded on every conversation you connect, and open the evidence behind each one.

Scoring every conversation is the measurement. The value shows up in the 1:1, when a team lead can point to the pattern and the conversations behind it.

Built example · runs every Monday by default

Agent Coaching Brief

Spend 1:1s coaching, not searching. Every Monday it reads each rep’s conversations from the week, matches them to your knowledge base and QA scores, and sends:

  • A brief per rep: the coaching points ranked by how often they came up and their customer impact, with the conversations to review
  • At least one strength per rep, with an example
  • A team rollup for the QA manager: the skills slipping across several reps at once
Sends to
SlackNotionConfluenceGoogle Drive

What is Rippit?

Rippit is a Growth Data Platform: it keeps enriching your customer conversations and other sources into structured data, connects them to CRM, billing and product data and each team’s own sources, and lets anyone set up agents on it, without needing an engineer to start. Connect with pre-built integrations or through our MCP, and ask from Rippit, Claude, ChatGPT or Cursor.

Support conversations (tickets, chat, call transcripts)AI agent chatsAccounts & billing (plans, ARR, renewals)Help-center & policy docsRelease notesand more

Build your first agent on your customer and business data.

Try Rippit free
or request demo

Keep reading

More from the Rippit Library.

View the library →