The Coverage Benchmark, Part 3: The Naive API
Mike Nucci·August 7, 2026What is the Coverage Benchmark?
Everyone with conversation data is trying to get insights out of it with Claude, and there are a lot of ways to do that: chat on a warehouse, agents, custom code, RAG, a purpose-built product. There are tradeoffs between them and we wanted to understand the scale. Is one approach 5% worse than another, or 50x worse? Cheaper by a dollar, or by $200? A minute slower, or an hour?
So we built a benchmark. Same 1,000 frozen customer calls, the same five questions, for every approach, scored on coverage, consistency, cost, and time.
Series note: This is Part 3 of the Coverage Benchmark. Part 1 tested Claude in the chat app on your warehouse; Part 2 gave it a real agent. Same 1,000 frozen calls, same five frozen questions. This time I stopped using anyone's product and wrote the code myself.
Result in brief
- Coverage: every run fit 99 of the 1,000 calls (9.7% of the text) into the context window, yet the answers opened “I analyzed all 1,000 transcripts…”.
- Control question (Q5): it found 1 of 46 calls, 2%, because 45 of them were in the 901 calls it never loaded.
- Consistency: 67% overlap, the highest so far, because it read the identical 99 calls every run.
- Time and cost: about 28 seconds and about $5.00 a question.
Our reading: a consistent answer can still come from a tenth of the evidence. Holding still says nothing about what was read.
The setup every engineer tries first
By Part 2 the harness looked like the problem: chat sampled the warehouse, and the agent looped but still sampled. So I cut out the middleman and sent the raw transcripts to Claude directly. No chat UI writing SQL, no agent spawning sub-agents, just the model and the data. That is Part 3, the naive API path.
One script, one API call. Pull the transcripts newest-first, stuff as many as fit into the context window, ask the frozen question, get an answer back. It's the first benchmark run where I set the coverage dial by hand instead of Claude setting it for me.
It maxed out at about 10%. This is the first test in the series that hits a hard limit, and Claude can’t talk its way past it.
It read 10%. That's the whole story.
The corpus is 1,000 transcripts — about 8.5M tokens. The context window, even with the 1M beta on, is about a million. Only roughly an eighth of the tokens can physically be in the room at once.
My script packs newest-first; on every run it pulled 99 of 1,000 calls - 9.7% of the text - and on every run it hit the wall (the first prompt came back over-length and the script backed off).
Before the model reasons about anything, 90% of the corpus has been dropped on the floor. Not down-weighted. Absent.
Here's the part that turns a limitation into a liability. The answers open like this:
- "I analyzed all 1,000 transcripts…"
- "Looking through all 1,000 call transcripts provided…"
Sometimes it kind of noticed. One run hedged "the payload actually contains ~90 transcripts".
Claude kept presenting its work as complete. A caveat about sample size is the kind of footnote a reader skips on the way to a confident answer.
Worse: even when it knew it had ~90 of 1,000, it had no way to know what was in the 901 it couldn't see. Knowing you're partial is not knowing what you're missing.

The control question proves it: 2%
Q5 is in the benchmark precisely because it has no wiggle room.
"Which calls mention Competitor X?" is a text search.
The key is mechanical and closed: 46 calls mention them.
In Part 1 chat found all 46; in Part 2 the agent found all 46. Both scored 100%, because both touched every call.
The naive API found one.
2% recall on the single question a first-year script should ace - while announcing it had read "all 1,000 call transcripts provided." It didn't miss those 45 calls because the model is weak at search. It missed them because 45 of the 46 were in the 901 calls that were never loaded. You cannot find what you never read.
If you want the coverage ceiling in one number, it's that: same model, same key as the arms that scored 100%, landing at 2% — entirely because of how much data fit.

Everything else is downstream of the 10%
Every other quirk this arm showed is the same fact wearing a different hat:
- It was the most consistent arm — build-in-house asked three times gave 67% membership overlap, beating the agent's 52% and chat's 24%. Not a virtue: my packing is deterministic, so it reads the identical 99 calls every run. It's steady the way a stopped clock is steady, and consistency alone would rank it the winner.
- On enterprise pricing it guessed. "Enterprise" means ARR > $50K, a column in the warehouse. With only raw text, it inferred scale from figures mentioned on the calls and admitted several accounts "could not be confirmed."
- On sentiment it refused — a defensible call that the data type doesn't support the question, but notable that chat answered the same question over 100% of the rows. Same question, three arms, three behaviors.
Where it wins, and why that's the danger
Speed, decisively. These runs averaged 28 seconds — against chat's 3:40 and the agent's 11:25. One call, one answer, no loop, no queue.
And that's exactly the trap: you paste, wait twenty seconds, and a clean, confident, well-formatted answer comes back headed "I analyzed all 1,000 transcripts." Nothing in the experience tells you it's 10%.
Cost is flat at about $5.00 a question because it always fills the same window, and that $5 buys a read of a tenth of your data, with no setting for "read the rest."
The scorecard so far

Build-in-house recall stays pending for every system: it’s scored against the answer key in a separate evaluation, not part of these results. Q5 recall is mechanical and final.
How to read the scorecard (this key appears in every part)
- Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
- Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
- Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
- Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. Scoring against it is a separate evaluation, not part of these results, so this row stays "pending" for every system, including ours.
- Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
- Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.
The takeaway
I couldn't push it past about 10%, because the corpus doesn't fit and there's no setting for "fit anyway." Every failure downstream is a shadow of that one fact. The confident "all 1,000." The 1-of-46 on the one question with a mechanical answer. The 67% consistency that looks like trust and is actually a stopped clock. None of it is the model being dumb. All of it is the model being asked to answer from a tenth of the evidence and not being told which tenth.
This is the wall the rest of the series is about getting over. There are only two honest ways across it: read everything the expensive brute-force way, or pre-compute a judgment once and reuse it. Everything left in the benchmark is a variation on those two.
What this means for your team
The risk for your team isn’t the 10%. It’s that the answer said “all 1,000.” A summary like that can go into a roadmap or pricing review with nothing on the page showing that 901 calls, including 45 of the 46 Competitor X mentions, were never read. Before an AI answer about your customers reaches a decision, ask how many conversations it actually opened; the series recap lists the questions to ask any vendor or in-house build.
Next up
Part 4 — API map-reduce
My own code again, but this time built to read all 1,000 transcripts, in full, by chunking them and consolidating. Real 100% coverage, no window games. The arm that answers the question this one raises: once you finally read everything, does the answer hold still?
Methodology appendix: Anthropic Messages API, Claude Opus 4.8, 1M-token context beta (context-1m-2025-08-07) enabled, max output 8,000 tokens. Transcripts pulled from the frozen BENCHMARK_CORPUS_1K, packed newest-first to the token budget; on every run the first packed prompt exceeded context and the script backed off 15% once, landing at 99 calls / ~997K input tokens. Cost computed at Opus list ($5/$25 per M in/out). Five frozen questions, Q1 asked three times in independent calls. Full run logs (R015–R021) available.
Keep reading
More from the Rippit Library.
The Agent Reads More, but Not the Same Way Twice
Claude Code read 95% on one question and 1% on another, averaged 11 minutes, and kept only half its account list on a re-ask.
What It Takes to Trust an AI Answer About Your Customers
The series in one page: which approach read everything, which gave the same list twice, and what each means for a board number or a renewal list.
Every Call Read, Every Answer Saved
Question-specific classification read every transcript, found 45 of 46 control mentions and 52 build-in-house accounts, and returns the same saved list on a re-ask.