The Coverage Benchmark, Part 4: Reading Everything
Mike Nucci·August 10, 2026What is the Coverage Benchmark?
Everyone with conversation data is trying to get insights out of it with Claude, and there are a lot of ways to do that: chat on a warehouse, agents, custom code, RAG, a purpose-built product. There are tradeoffs between them and we wanted to understand the scale. Is one approach 5% worse than another, or 50x worse? Cheaper by a dollar, or by $200? A minute slower, or an hour?
So we built a benchmark. Same 1,000 frozen customer calls, the same five questions, for every approach, scored on coverage, consistency, cost, and time. Those insights end up in renewal lists, forecasts and board numbers. This part asks whether reading everything is enough to make them hold still.
Series note: This is Part 4 of the Coverage Benchmark. Part 1 tested Claude in the chat app, Part 2 tested Claude as an agent, and Part 3 tested it as one naive API call. Every one of them sampled a fraction of the data. This time I wrote code that reads all 1,000 transcripts in full. This is the arm where I stop worrying about coverage and start worrying about something worse.
Result in brief
- Coverage: 100% of every transcript on all seven runs, verified against batch checkpoints.
- Cost and time: about $61 a question (about $183 for the three build-in-house runs) and about two minutes.
- Consistency: 42, 46 and 42 accounts; 73 named across the three runs, only 16 in all three, 22% overlap.
- Control question (Q5): the LLM read found 36 of 46 calls, 78%; a plain text scan finds all 46.
Our reading: count stability is not result stability. Totals within four of each other sat on lists that shared fewer than a quarter of their names.
The honest way to read everything
Every arm so far has read a slice of the conversations.
Chat sampled the warehouse (~10%), the agent looped but still sampled (~31%), the naive API physically couldn't fit more than 10% in the window.
So the obvious move: stop letting anything sample. Read 100% of the conversations.
That's a specific piece of code - map-reduce. I chunk the 1,000 transcripts into 50 batches of 20, and for each batch I make one API call that reads every word and extracts structured findings (which accounts, which quotes). That's the map.
Then one final call consolidates all 50 batches' findings into the answer: the reduce. Every transcript is read exactly once, in full. No keyword filter, no truncation.
This is what "just read everything" actually looks like when you build it, and it's the honest baseline the series needed.
It worked.
And then it handed me the most unsettling result in the benchmark so far.
It read everything
Same corpus, same five frozen questions. Opus 4.8, 50 batches per run, every call read once.


Answering one question costs about $61.
That's the real price of reading everything once - roughly twelve times the naive call's $5, because it actually processes six-to-eight million more tokens instead of throwing them away.
Fine. Maybe I would pay $61 for a great, accurate, trustworthy answer.
The problem is I didn't get one.

Then I asked the same question three times
Build-in-house, three independent runs, 100% coverage each. The counts looked somewhat reassuring: 42, 46, 42 accounts. If I'd stopped at the totals I'd have called it stable and shipped it.
Then I looked at what accounts were on each list.
Across the three runs, 73 different accounts got named - and only 16 appeared in all three!!
That's 22% overlap. Reading every word of all 1,000 transcripts, three times, gave me three materially different customer lists - lower agreement than the chat app managed (24%) while reading a tenth of the data.
So I’d spend about $61 to get an answer I can’t trust to hold still.
16 of 73 accounts appear in all three - 22%

Why reading everything didn't fix it
This is the hinge of the whole series, so I want to be precise about the mechanism.
Coverage guarantees every call is seen. It guarantees nothing about whether the same call gets judged the same way twice. "Is this customer considering building in-house?" isn't a keyword, it's a judgment.
A marginal account that reads as a "maybe" gets included in run 1, dropped in run 3. Multiply that across 1,000 calls and 50 batches and you get a union of 73 with a stable-looking middle and a churning edge.
So I spent ~$61 on each run to get full coverage (and ~$183 total on the 3 runs), and got three different, unreliable answers. The issue was not how much got read. Parts 1–3 failed by reading too little. This arm reads everything and still fails, which means more reading wasn’t the fix.

Everything else is downstream of that
- The control question, again. Even reading every word, the LLM's extraction found 36 of 46 Competitor-X mentions (78%). Chat and the agent found all 46 - because they ran a dumb text scan, and for a pure keyword a deterministic scan beats an LLM that has to re-notice the word in every batch. Full read is not full extraction.
- You pay the whole bill every time. $61 buys one answer. Ask again next week and it's another $61, because nothing was stored - the read is spent the moment it's done. Every question is a fresh full-price re-derivation.
- Structured questions still miss. Enterprise pricing returned only 11 findings: "ARR > $50K" is a column, and there are no columns here, only dialogue. Even a complete read can't filter a field that was never in the text.
Where it genuinely wins
This is the first arm that actually reads 100%, and on aggregate/judgment questions - churn reasons, sentiment - it considers evidence the sampling arms never touched. If what you need is "every call weighed once," this is pretty good.
It's a legitimate research instrument. It is not a system you'd wire to a recurring dashboard unless you want to go broke.
The scorecard so far

Recall on build-in-house stays pending for every system; it’s scored in a separate evaluation, not part of these results. Q5 recall is mechanical for the sampling arms (text scan) and LLM-read for map-reduce. Noted because the difference is itself a finding.
How to read the scorecard (this key appears in every part)
- Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
- Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
- Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
- Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. Scoring against it is a separate evaluation, not part of these results, so this row stays "pending" for every system, including ours.
- Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
- Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.
The takeaway
Map-reduce is the honest brute-force answer to "just read everything," and it delivers exactly what it promises on coverage: 100%, every run, verified.
It also exposed a bigger problem: the same question on the same data does not give the same answer. Three full reads of my data gave me three different customer lists.
The problem isn’t how much gets read.
It's that every question discards the judgment and rebuilds it from scratch, which is slow and expensive.
Which points to the only real fix. I'm already at 100% coverage; I can't read more. The move is to read once, make the judgment once, and keep it. So the next question is a lookup against a stored answer, not a $61 re-derivation that lands somewhere new. That's the entire premise of the back half of this series.
What this means for your team
What the test showed: full coverage, three different lists. What that means in practice: an account list is only useful if it holds still. If an account considering building in-house is on one run’s list and off the next, nobody knows whether to plan the save call, and at about $61 a question, paid again each time a team asks, rechecking doesn’t scale. The fix this series points to is storing each call’s classification and the evidence behind it, then connecting it to the account’s ARR and renewal date, so CS, Sales and Finance work from the same underlying signals. The series recap explains why persisted classifications held up best.
Renewal Risk Watch
An example, not a packaged product: describe it in plain language and it could run each week. It judges every support conversation and sales and CS call once, saves the verdict per account so the list holds on a re-ask, joins it to CRM, usage and billing, and sends:
- Every renewal whose risk changed, such as Low to High, with the reasons: usage down, escalations piling up, a champion gone, no next step on the renewal
- A customer quote and links to the tickets, calls and usage behind each flag
- A suggested next step, such as an exec check-in before the QBR
- A leadership roll-up of the renewals that changed risk and the ARR at stake
Next up
Part 5 — the Batch API
The identical map-reduce, submitted as a batch job: less than half the cost, if you can wait minutes for the result.
Methodology appendix: Anthropic Messages API, Claude Opus 4.8, map-reduce over the frozen BENCHMARK_CORPUS_1K - 50 batches of 20 transcripts, each read in full (max 4,000 output tokens per map call), one reduce call to consolidate. Coverage is 100% by construction and verified against batch checkpoints. Cost at Opus list ($5/$25 per M in/out); three early runs were logged at the deprecated $15/$75 rate and are corrected by exact factor of 3. Five frozen questions, Q1 asked three times in independent runs. Full run logs (R022, R023, R023B, R024–R027) and per-batch checkpoints available.
Keep reading
More from the Rippit Library.
“I Analyzed All 1,000.” It Read 99.
One large API call read about a tenth of the corpus, found 1 of 46 competitor mentions, and still claimed full coverage.
What It Takes to Trust an AI Answer About Your Customers
The series in one page: which approach read everything, which gave the same list twice, and what each means for a board number or a renewal list.
Every Call Read, Every Answer Saved
Question-specific classification read every transcript, found 45 of 46 control mentions and 52 build-in-house accounts, and returns the same saved list on a re-ask.