The Coverage Benchmark, Part 8: Rippit

Part 8, the finale. Seven arms in, we've tried every way to read the data - a little, all of it, a chosen slice, and a pre-computed summary. This is the one the whole benchmark was built to test: our own product, Rippit. I ran it last on purpose, so nothing it did could bias how I graded everything else. Same 1,000 calls, same five questions, same scoring.

Rippit - why it ran last and why it’s different

I did the Rippit test last so that this benchmark report is unbiased.

The answer key, the coverage math, the consistency test - all of it was locked in against the seven other tests before we used Rippit at all.

If it looks good here, it's not because I graded it gently. I graded it under the same rules used for the other tests.

Here's what makes it different mechanically:

The other arms are all one-shot: you ask, and each re-reads the data from scratch, every time, re-deriving the answer and paying for the answer again.

Rippit is smarter and more like a harness.

You ask, "who's considering building in-house?" and it first goes and looks for the answer: has this question already been run on these specific conversations I’m analyzing? If it has, you get the answer right away without it costing you any more money to read the full transcript.

If it hasn't, it kicks off a cheap, quick job that reads 100% of every transcript, classifies each of the conversations against the question, and writes the verdict back as a column, which it saves in a table.

Either way you end up with a saved column you can query, audit, and reuse. It's the "find the answer if it's there, and if it isn't, read everything once to get it" idea that Parts 3–7 kept circling without doing.

Q5 is the question with a mechanical answer - 46 calls mention Competitor X, and a plain text search finds all 46.

Every prior LLM-based test fell short of that: RAG 28%, batch 54%, map-reduce 78%, and pre-computed enrichment just 50%, because the generic summaries had already thrown away half the mentions.

Rippit, asked that exact question as its own read of every transcript, caught 45 of 46 - 98%.

Part 7 hit only 50% because it reused a summary built for something else. Read every transcript for the question you're actually asking, and you get 98%.

Figure 1. The mechanical control question across the whole series. The text-scan arms hit 100% for free; among the arms that actually reason, only Rippit - reading every transcript for this exact question - comes near it. Part 7's generic summary got half.

Same story on Q1. Rippit flagged 52 accounts, every one carrying a customer quote.

RAG found 0, the naive call 2–3, pre-computed enrichment 12. It caught Customer X (already moved to an internal tool), plus dozens more the sampling and retrieval arms never had a chance at - because it wasn't guessing from similarity or a truncated read, it had classified every call against the question.

The cost isn't a typo. Reading 100% of the transcripts is the thing map-reduce paid $61 a question for.

Rippit did the same full read for about six cents - because the per-call classification is small and cheap. 

It's the cheapest arm in the entire benchmark and the one that reads the most.

And to be clear - this is the same cost Rippit’s customers pay.

The input and output token cost advertised by the model providers.

No mark-up.

Now the consistency row - the same question, three times, do you get the same answer? For every other arm, asking again means computing again: chat re-queries, map-reduce re-reads all 1,000, the API re-runs.

So you get a fresh, different answer each time, and they score 24%, 22%, 29%. Asking a probabilistic model the same question multiple times on the same set of data - inherently means you’ll get different answers, which instills distrust in the answers you’re getting.

Rippit keeps the column. Ask "who's considering building in-house?" once, and it classifies all 1,000 calls and saves the result.

Ask Rippit again, and it will go find the saved columns and give you the same answer it did before.

Rippit is the only test that could get 100% recall - because it's the only one that saves the answer instead of re-trying.

100% recall means that if you and someone else ask the same question of the same data - you’ll get the same answer. It doesn’t mean the answer is 100% right.

Every one of the answers on each of the 1,000 conversations is still a probabilistic judgment call that an LLM makes.

So when I force Rippit to re-classify from scratch three separate times - the passes agree only about 52%.

It read the same complete conversation of every call, all three times. The variance comes down to genuinely hard cases - sometimes someone is highlighting it as a possibility or posing it as an option they’ve heard of people doing or the customer’s intent could be something else entirely.

The LLM is trying to make a similar judgment that I did reading the transcripts. The clear-cut accounts are rock solid every single time; only the real judgment calls move.

And that's a variance you can actually do something about - precisely because it read the whole transcript and wrote down its reasoning and the supporting quote for every call. You open the column, go to the twenty borderline ones, read the evidence it based each call on, and decide yourself.

*Coverage of 100% for map-reduce/batch and Rippit is a real full-transcript read; for Claude+enriched its 100% of persisted judgments (a summary, not raw text). Q1 recall closes for every arm in the final report.

†Rippit consistency is 100% because a re-asked question returns the saved column, not a re-computation - the reproducibility of the metric measures, and the only arm that has it. Forced to re-classify from scratch, its underlying judgment agrees ~52% (the fuzzy-margin wobble), which is why the column being openable and auditable is the point, not the score.

The build-in-house row is the last thing outstanding. The key needed every system's candidates before it could close, and Rippit was the last one. It closes in the wrap up report.

• Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
• Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
• Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
• Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. It closes only after all systems run - so this row stays "pending" until the final report, for every system including ours.
• Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
• Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

Eight tests, 1,000 conversations, five questions.

Figure 2. Every arm with a clean marginal cost, plotted by what it reads against what it costs. The two ways to hit full coverage on raw text - map-reduce and batch - sit on the expensive right. Rippit is alone in the top-left: everything read, for the least money. (C1's 100% is coverage of summaries, not raw text.)

The tests that didn’t read 100% of the conversations (chat, naive, RAG) miss most of the data and aren’t transparent about not reading 100% of the conversations.

Reading 100% of each conversation (map-reduce, batch) costs a fortune with Snowflake Cortex and still gives you a different answer every time.

Reading a pre-built summary (Part 7) is fast and cheap, but it's capped by whatever the summary happens to keep.

The only thing that broke the pattern was reading 100% of each transcript using Rippit as the harness.

That's what got Q5 from 50% to 98% and Q1 from single digits to dozens, for six cents.

So there are a couple of lessons - Cortex is expensive, Cortex is slow, you have to save answers to keep costs even lower and performance even higher.

Next up

The final report. Every scorecard so far has had one empty row. The final report fills it, for all eight arms, and ties the series together.

Methodology appendix: Claude Opus 4.8, 1M-token context beta enabled, one call per question over the pre-computed enrichment for all 1,000 calls (columns: sentiment, churn risk, competitors mentioned, topics, contact reason, product areas, key words, summary, opportunity amount, issue resolved, urgency; ~737K input tokens). No raw transcripts in context. Cost at Opus list ($5/$25 per M in/out); the enrichment build is a prior, sunk cost, noted but not counted as marginal. Five frozen questions, Q1 asked three times. Coverage reported as 100% of persisted judgments and footnoted as such. Full run logs (R042–R048) available.