With Rippit (formerly MaestroQA), anyone can build a QA agent that scores 100% of customer conversations and connects quality results to customer insights. This page does the arithmetic; for the case for scoring every conversation, see how to QA 100% of support conversations.
Key takeaways
- 299 random reviews give about a 95% chance of seeing a 1% issue once; 100 reviews give about 63%.
- Halve the issue rate and you roughly double the sample you need.
- “We review 2%” is a different policy at every volume: 40 reviews at 2,000 conversations a month, 1,000 at 50,000.
- Detecting an issue and measuring its rate are different jobs; measuring takes roughly five times the sample.
- When every conversation is scored, finding rare failures becomes a filter, not a probability.
What’s the formula for catching a rare issue?
Suppose an issue occurs in a fraction p of conversations and you randomly review n of them. The probability that you see it at least once is:
Turn it around to get the sample size for a detection probability you choose, rounding up:
For an issue that occurs in 1% of conversations:
| Conversations reviewed (random) | Chance of catching a 1% issue at least once |
|---|---|
| 25 | 22% |
| 50 | 39% |
| 100 | 63% |
| 150 | 78% |
| 230 | 90% |
| 299 | 95% |
| 459 | 99% |
The surprising number is 100. It feels like a meaningful QA sample, but there is still roughly a one-in-three chance that a 1% failure never appears in it. (The same math explains why AI assistants that sample your conversations miss rare problems.)
How many reviews do you need at other issue rates?
Random reviews needed to see an issue at least once (copy this into your QA plan):
| Issue rate | 90% chance | 95% chance | 99% chance |
|---|---|---|---|
| 5% | 45 | 59 | 90 |
| 1% | 230 | 299 | 459 |
| 0.5% | 460 | 598 | 919 |
| 0.1% | 2,302 | 2,995 | 4,603 |
Computed from the formula above, not from customer data.
Why doesn’t “we review 2% of tickets” tell you much?
Human QA typically reviews 1–3% of interactions (AI agent monitoring playbook), so many programs define coverage as a percentage. But a percentage is not a detection probability:
| Monthly volume | 2% sample | Chance of catching a 1% issue | Chance of catching a 0.1% issue |
|---|---|---|---|
| 2,000 | 40 | 33% | 4% |
| 10,000 | 200 | 87% | 18% |
| 50,000 | 1,000 | 99.996% | 63% |
So “we review 2%” is not one QA policy: at low volume it barely detects a 1% failure; at high volume it is close to exhaustive. (Our 100% QA guide works through the 10,000 row for more failure rates.)
Is detecting an issue the same as measuring it?
No. Finding a 1% issue in 299 random reviews does not let you say “the rate fell from 1.2% to 0.8% this quarter.” That is an estimation question.
A standard planning calculation, n = 1.96² × p(1 − p) ÷ margin², gives about 1,521 reviews to estimate a rate near 1% within ±0.5 percentage points at 95% confidence. Even then, an exact interval around 15 findings in 1,521 runs from about 0.55% to 1.62%, so treat roughly 1,500 as a floor.
A middle step is the three-sightings rule: see something three times before calling it a pattern. For a 1% issue you expect three sightings at about 300 reviews, but a 95% chance of three or more takes about 628.
Detection asks: Did this happen? Estimation asks: How often? Coverage asks: In which conversations? One sampling policy rarely answers all three.
Why do real QA queues make the math messier?
The formula assumes a random sample. Real QA programs rarely draw one: reviewers choose escalations, low-CSAT interactions or new agents, and failures cluster in one team, workflow or customer segment. Oversampling a high-risk queue can make you more likely to find a problem, but the formula no longer tells you your detection odds, and population-wide rates from that sample can mislead.
How should you design a QA sampling policy?
Start with the problem, not a percentage:
- Choose the failure: a compliance failure, an incorrect refund, a missed escalation.
- Set the prevalence you can’t afford to miss: 5%, 1% or 0.1%.
- Choose a detection probability, such as 95%, and look up the sample in the table.
- Size measurement separately. It will be much larger.
- Document targeted sampling. It is useful context, not something to hide.
What changes when you score every conversation?
You stop asking “did our reviewers happen to find one?” and ask “which conversations triggered this criterion?” A 0.5% failure across 50,000 monthly conversations is 250 conversations; a 2% sample would expect to see about five. Scoring everything finds all 250, and reviewers move to calibration, exceptions and coaching (where QA reviewers go).
In Rippit, that QA agent is an Agent App: you describe it once in plain language (was the problem understood, the right next step given, the issue resolved?), test it, and it runs on a schedule, delivering results to Slack or email, with the conversations behind every score (use cases).
Connect your customer conversation hubs to Rippit in one click—including Zendesk, Intercom, Gong and more—then tell the agent what to check across 100% of your connected conversations (how it works, integrations). Through our MCP, the agent can also read, write and update data across your broader stack, including Salesforce, HubSpot, Slack, Notion, Guru, Jira, Snowflake, and more.
Sampling isn’t inherently bad: a modest random sample can detect a common problem. The first question for any QA program is: are we trying to detect a problem, measure it, or find every instance of it? More in the Coverage Benchmark Series.
FAQ
How many tickets do I need to review to catch a 1% issue?
About 299 random conversations for a 95% chance of seeing it at least once, and about 459 for 99%. One hundred gives about 63%.
Why isn’t reviewing 2% of tickets enough?
Because 2% is a sampling rate, not a detection probability: 40 reviews at 2,000 conversations catch a 1% issue about a third of the time. To make the case to a director, name the failure rate you can’t afford to miss and your sample’s chance of catching it.
How many reviews do I need to measure a 1% issue rate?
About 1,521 random reviews to estimate a rate near 1% within ±0.5 percentage points at 95% confidence, using a standard planning formula.
Does automated QA eliminate the need for human reviewers?
No. Reviewers move from searching a sample to calibration, exceptions, high-severity cases and coaching.
Build your first agent on your own customer conversations. Free, no credit card, no engineer needed.
Want a more personalized walkthrough? Request a demo.
.webp)
