research

The Coverage Benchmark, Part 4: Reading Everything

Mike NucciMike NucciAugust 10, 2026

What is the Coverage Benchmark?

Everyone with conversation data is trying to get insights out of it with Claude, and there are a lot of ways to do that: chat on a warehouse, agents, custom code, RAG, a purpose-built product. There are tradeoffs between them and we wanted to understand the scale. Is one approach 5% worse than another, or 50x worse? Cheaper by a dollar, or by $200? A minute slower, or an hour?

So we built a benchmark. Same 1,000 frozen customer calls, the same five questions, for every approach, scored on coverage, consistency, cost, and time. Those insights end up in renewal lists, forecasts and board numbers. This part asks whether reading everything is enough to make them hold still.

Series note: This is Part 4 of the Coverage Benchmark. Part 1 tested Claude in the chat app, Part 2 tested Claude as an agent, and Part 3 tested it as one naive API call. Every one of them sampled a fraction of the data. This time I wrote code that reads all 1,000 transcripts in full. This is the arm where I stop worrying about coverage and start worrying about something worse.

Result in brief

  • Coverage: 100% of every transcript on all seven runs, verified against batch checkpoints.
  • Cost and time: about $61 a question (about $183 for the three build-in-house runs) and about two minutes.
  • Consistency: 42, 46 and 42 accounts; 73 named across the three runs, only 16 in all three, 22% overlap.
  • Control question (Q5): the LLM read found 36 of 46 calls, 78%; a plain text scan finds all 46.

Our reading: count stability is not result stability. Totals within four of each other sat on lists that shared fewer than a quarter of their names.

The honest way to read everything

Every arm so far has read a slice of the conversations.

Chat sampled the warehouse (~10%), the agent looped but still sampled (~31%), the naive API physically couldn't fit more than 10% in the window.

So the obvious move: stop letting anything sample. Read 100% of the conversations.

That's a specific piece of code - map-reduce. I chunk the 1,000 transcripts into 50 batches of 20, and for each batch I make one API call that reads every word and extracts structured findings (which accounts, which quotes). That's the map.

Then one final call consolidates all 50 batches' findings into the answer: the reduce. Every transcript is read exactly once, in full. No keyword filter, no truncation.

This is what "just read everything" actually looks like when you build it, and it's the honest baseline the series needed.

It worked.

And then it handed me the most unsettling result in the benchmark so far.

It read everything

Same corpus, same five frozen questions. Opus 4.8, 50 batches per run, every call read once.

Table showing five questions, map-reduce results, 100% coverage, and various completion times.
Bar chart showing Map-reduce (B2) covers 100% of corpus, Agent (A2) 31%, Chat (A1) and Naive API (B1) each 10%.
Figure 1. The first arm that actually reads everything: chat 10%, agent 31%, naive API 10%, map-reduce 100%.

Answering one question costs about $61.

That's the real price of reading everything once - roughly twelve times the naive call's $5, because it actually processes six-to-eight million more tokens instead of throwing them away.

Fine. Maybe I would pay $61 for a great, accurate, trustworthy answer.

The problem is I didn't get one.

Bar chart comparing $5 cost for Naive API with $61 for Map-reduce at 100% coverage API tokens.
Figure 2. Same API billing basis: $5 to read 10%, $61 to read 100% - and the bill lands again on every question, because nothing is stored.

Then I asked the same question three times

Build-in-house, three independent runs, 100% coverage each. The counts looked somewhat reassuring: 42, 46, 42 accounts. If I'd stopped at the totals I'd have called it stable and shipped it.

Then I looked at what accounts were on each list.

Across the three runs, 73 different accounts got named - and only 16 appeared in all three!!

That's 22% overlap. Reading every word of all 1,000 transcripts, three times, gave me three materially different customer lists - lower agreement than the chat app managed (24%) while reading a tenth of the data.

So I’d spend about $61 to get an answer I can’t trust to hold still.

16 of 73 accounts appear in all three - 22%

Bar graph shows 42, 46, 42 accounts named in three runs; 16 appear in all three, 22% of 73 distinct accounts.
Figure 3. Same question, same full data, three times. The total holds; the list doesn't.

Why reading everything didn't fix it

This is the hinge of the whole series, so I want to be precise about the mechanism.

Coverage guarantees every call is seen. It guarantees nothing about whether the same call gets judged the same way twice. "Is this customer considering building in-house?" isn't a keyword, it's a judgment.

A marginal account that reads as a "maybe" gets included in run 1, dropped in run 3. Multiply that across 1,000 calls and 50 batches and you get a union of 73 with a stable-looking middle and a churning edge.

So I spent ~$61 on each run to get full coverage (and ~$183 total on the 3 runs), and got three different, unreliable answers. The issue was not how much got read. Parts 1–3 failed by reading too little. This arm reads everything and still fails, which means more reading wasn’t the fix.

Bar chart comparing coverage and consistency for Chat, Agent, Naive API, and Map-reduce, with Map-reduce highest coverage.
Figure 4. Across every arm scored so far, the one reading the most (map-reduce, 100%) agrees with itself the least (22%). Coverage and consistency are independent axes - buying one does not buy the other.

Everything else is downstream of that

  • The control question, again. Even reading every word, the LLM's extraction found 36 of 46 Competitor-X mentions (78%). Chat and the agent found all 46 - because they ran a dumb text scan, and for a pure keyword a deterministic scan beats an LLM that has to re-notice the word in every batch. Full read is not full extraction.
  • You pay the whole bill every time. $61 buys one answer. Ask again next week and it's another $61, because nothing was stored - the read is spent the moment it's done. Every question is a fresh full-price re-derivation.
  • Structured questions still miss. Enterprise pricing returned only 11 findings: "ARR > $50K" is a column, and there are no columns here, only dialogue. Even a complete read can't filter a field that was never in the text.

Where it genuinely wins

This is the first arm that actually reads 100%, and on aggregate/judgment questions - churn reasons, sentiment - it considers evidence the sampling arms never touched. If what you need is "every call weighed once," this is pretty good.

It's a legitimate research instrument. It is not a system you'd wire to a recurring dashboard unless you want to go broke.

The scorecard so far

Table comparing coverage, consistency, recall, cost, and time per question across different API and agent methods.

Recall on build-in-house stays pending for every system; it’s scored in a separate evaluation, not part of these results. Q5 recall is mechanical for the sampling arms (text scan) and LLM-read for map-reduce. Noted because the difference is itself a finding.

How to read the scorecard (this key appears in every part)

  • Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
  • Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
  • Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
  • Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. Scoring against it is a separate evaluation, not part of these results, so this row stays "pending" for every system, including ours.
  • Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
  • Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

The takeaway

Map-reduce is the honest brute-force answer to "just read everything," and it delivers exactly what it promises on coverage: 100%, every run, verified.

It also exposed a bigger problem: the same question on the same data does not give the same answer. Three full reads of my data gave me three different customer lists.
The problem isn’t how much gets read.

It's that every question discards the judgment and rebuilds it from scratch, which is slow and expensive.

Which points to the only real fix. I'm already at 100% coverage; I can't read more. The move is to read once, make the judgment once, and keep it. So the next question is a lookup against a stored answer, not a $61 re-derivation that lands somewhere new. That's the entire premise of the back half of this series.

What this means for your team

What the test showed: full coverage, three different lists. What that means in practice: an account list is only useful if it holds still. If an account considering building in-house is on one run’s list and off the next, nobody knows whether to plan the save call, and at about $61 a question, paid again each time a team asks, rechecking doesn’t scale. The fix this series points to is storing each call’s classification and the evidence behind it, then connecting it to the account’s ARR and renewal date, so CS, Sales and Finance work from the same underlying signals. The series recap explains why persisted classifications held up best.

An example of an agent you could build

Renewal Risk Watch

An example, not a packaged product: describe it in plain language and it could run each week. It judges every support conversation and sales and CS call once, saves the verdict per account so the list holds on a re-ask, joins it to CRM, usage and billing, and sends:

  • Every renewal whose risk changed, such as Low to High, with the reasons: usage down, escalations piling up, a champion gone, no next step on the renewal
  • A customer quote and links to the tickets, calls and usage behind each flag
  • A suggested next step, such as an exec check-in before the QBR
  • A leadership roll-up of the renewals that changed risk and the ARR at stake

Next up

Part 5 — the Batch API

The identical map-reduce, submitted as a batch job: less than half the cost, if you can wait minutes for the result.

Methodology appendix: Anthropic Messages API, Claude Opus 4.8, map-reduce over the frozen BENCHMARK_CORPUS_1K - 50 batches of 20 transcripts, each read in full (max 4,000 output tokens per map call), one reduce call to consolidate. Coverage is 100% by construction and verified against batch checkpoints. Cost at Opus list ($5/$25 per M in/out); three early runs were logged at the deprecated $15/$75 rate and are corrected by exact factor of 3. Five frozen questions, Q1 asked three times in independent runs. Full run logs (R022, R023, R023B, R024–R027) and per-batch checkpoints available.

Share this post

Keep reading

More from the Rippit Library.

View the library →