The Coverage Benchmark, Part 4: Reading Everything

Series note: This is Part 4 of the Coverage Benchmark. Parts 1 tested Claude in the chat app, Part 2 tested Claude as an agent, and Part 3 as one naive API call. Every one of them sampled a fraction of the data. This time I wrote code that reads all 1,000 transcripts in full. This is the arm where I stop worrying about coverage and start worrying about something worse.

The honest way to read everything

Every arm so far has read a slice of the conversations.

Chat sampled the warehouse (~10%), the agent looped but still sampled (~31%), the naive API physically couldn't fit more than 10% in the window.

So the obvious move: stop letting anything sample. Read 100% of 100% of the conversations.

That's a specific piece of code - map-reduce. I chunk the 1,000 transcripts into 50 batches of 20, and for each batch I make one API call that reads every word and extracts structured findings (which accounts, which quotes). That's the map.

Then one final call consolidates all 50 batches' findings into the answer: the reduce. Every transcript is read exactly once, in full. No keyword filter, no truncation.

This is what "just read everything" actually looks like when you build it, and it's the honest baseline the series needed.

It worked. Coverage came out at 100% on all seven runs — verified, not claimed.

And then it handed me the most unsettling result in the benchmark so far.

Same corpus, same five frozen questions. Opus 4.8, 50 batches per run, every call read once.

My script packs newest-first; on every run it pulled 99 of 1,000 calls - 9.7% of the text - and on every run it actually hit the wall (the first prompt came back over-length and the script backed off).

Figure 1. The first arm that actually reads everything: chat 10%, agent 31%, naive API 10%, map-reduce 100%.

But answering 1 single question ain’t cheap! It costs ~$61 a question!!!

That's the real price of reading everything once - roughly six times the naive call's $5, because it actually processes six-to-eight million more tokens instead of throwing them away.

Fine. Maybe I would pay $61 for a great, accurate, trustworthy answer.

The problem is I didn't get one.

Figure 2. Same API billing basis: $5 to read 10%, $61 to read 100% - and the bill lands again on every question, because nothing is stored.

Build-in-house, three independent runs, 100% coverage each. The counts looked somewhat reassuring: 42, 46, 42 accounts. If I'd stopped at the totals I'd have called it stable and shipped it.

Then I looked at what accounts were on each list.

Across the three runs, 73 different accounts got named - and only 16 appeared in all three!!

That's 22% overlap. Reading every word of all 1,000 transcripts, three times, gave me three materially different customer lists - lower agreement than the chat app managed (24%) while reading a tenth of the data.

So in theory I’d spend ~$60 bucks to get an answer that I can’t trust and turns out to be wrong!

16 of 73 accounts appear in all three - 22%

Figure 3. Same question, same full data, three times. The total holds; the list doesn't.

This is the hinge of the whole series, so I want to be precise about the mechanism.

Coverage guarantees every call is seen. It guarantees nothing about whether the same call gets judged the same way twice. "Is this customer considering building in-house?" isn't a keyword, it's a judgment.

A marginal account that reads as a "maybe" gets included in run 1, dropped in run 3. Multiply that across 1,000 calls and 50 batches and you get a union of 73 with a stable-looking middle and a churning edge.

So I spent ~$61 on each run to get full coverage (and ~$183 total on the 3 runs), and got three different, unreliable answers. The issue was not how much got read. Parts 1–3 failed by reading too little. This arm reads everything and still fails, which means more reading wasn’t the fix.

Figure 4. Across every arm scored so far, the one reading the most (map-reduce, 100%) agrees with itself the least (22%). Coverage and consistency are independent axes - buying one does not buy the other.

• The control question, again. Even reading every word, the LLM's extraction found 36 of 46 Competitor-X mentions (78%). Chat and the agent found all 46 - because they ran a dumb text scan, and for a pure keyword a deterministic scan beats an LLM that has to re-notice the word in every batch. Full read is not full extraction.
• You pay the whole bill every time. $61 buys one answer. Ask again next week and it's another $61, because nothing was stored - the read is spent the moment it's done. Every question is a fresh full-price re-derivation.
• Structured questions still miss. Enterprise pricing returned only 11 findings: "ARR > $50K" is a column, and there are no columns here, only dialogue. Even a complete read can't filter a field that was never in the text.

This is the first arm that actually reads 100%, and on aggregate/judgment questions - churn reasons, sentiment - it considers evidence the sampling arms never touched. If what you need is "every call weighed once," this is pretty good.

It's a legitimate research instrument. It is not a system you'd wire to a recurring dashboard unless you want to go broke.

Recall on build-in-house stays pending for every system until the key closes after all systems run. Q5 recall is mechanical for the sampling arms (text scan) and LLM-read for map-reduce. Noted because the difference is itself a finding.

• Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
• Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
• Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
• Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. It closes only after all systems run - so this row stays "pending" until the final report, for every system including ours.
• Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
• Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

Map-reduce is the honest brute-force answer to "just read everything," and it delivers exactly what it promises on coverage: 100%, every run, verified.

It also exposed a pretty significant issue: you can’t reproduce the same answer asked on the same data. Three full reads of my data gave me three different customer lists.
The problem was not only how much gets read.

It's that the judgment is discarded and rebuilt from scratch on every question, and that’s too slow and too expensive.

Which points to the only real fix. I'm already at 100% coverage; I can't read more. The move is to read once, make the judgment once, and keep it. So the next question is a lookup against a stored answer, not a $61 re-derivation that lands somewhere new. That's the entire premise of the back half of this series.

Part 5 — the Batch API

The identical map-reduce, submitted as a batch job: about half the cost, if you can trade minutes for hours.

Methodology appendix: Anthropic Messages API, Claude Opus 4.8, map-reduce over the frozen BENCHMARK_CORPUS_1K - 50 batches of 20 transcripts, each read in full (max 4,000 output tokens per map call), one reduce call to consolidate. Coverage is 100% by construction and verified against batch checkpoints. Cost at Opus list ($5/$25 per M in/out); three early runs were logged at the deprecated $15/$75 rate and are corrected by exact factor of 3. Five frozen questions, Q1 asked three times in independent runs. Full run logs (R022, R023, R023B, R024–R027) and per-batch checkpoints available.