The Coverage Benchmark, Part 3: The Naive API

Series note: This is Part 3 of the Coverage Benchmark. Part 1 tested Claude in the chat app on your warehouse; Part 2 gave it a real agent. Same 1,000 frozen calls, same five frozen questions. This time I stopped using anyone's product and wrote the code myself.

The setup every engineer tries first

By Part 2 it seemed obvious that we needed to try the API route.

Chat sampled the warehouse; the agent looped but still sampled. You naturally think that the harness is the problem.

Pull the raw transcripts and send them to Claude directly.

Cut out the middleman. No chat UI deciding to write SQL, no agent deciding to spawn sub-agents.

Just the model and the data.

So that's Part 3: the naive API path.

One script, one API call. Pull the transcripts newest-first, stuff as many as fit into the context window, ask the frozen question, get an answer back. It's the first benchmark run where I set the coverage dial by hand instead of Claude setting it for me.

It maxed out at about 10%. And this is the first test in the series that runs into an immovable object, unfortunately Claude isn’t any better than an unstoppable force.

The corpus is 1,000 transcripts — about 8.5M tokens. The context window, even with the 1M beta on, is about a million. Only an eighth of the data can physically be in the room at once.

My script packs newest-first; on every run it pulled 99 of 1,000 calls - 9.7% of the text - and on every run it actually hit the wall (the first prompt came back over-length and the script backed off).

Before the model reasons about anything, 90% of the corpus has been dropped on the floor. Not down-weighted. Absent.

Here's the part that turns a limitation into a liability. The answers open like this:

• "I analyzed all 1,000 transcripts…"
• "Looking through all 1,000 call transcripts provided…"

Sometimes it kind of noticed. One run hedged "the payload actually contains ~90 transcripts".

Claude kept framing its work as if it was complete, and a lie about sample size is a footnote a reader skips on the way to the confident answer.

Worse: even when it knew it had ~90 of 1,000, it had no way to know what was in the 901 it couldn't see. Knowing you're partial is not knowing what you're missing.

Figure 1. One call, one window. Nine of every ten conversations never entered the model.

Q5 is in the benchmark precisely because it has no wiggle room.

"Which calls mention Competitor X?" is a text search.

The key is mechanical and closed: 46 calls mention them.

In Part 1 chat found all 46; in Part 2 the agent found all 46. Both scored 100%, because both touched every call.

The naive API found one.

2% recall on the single question a first-year script should ace - while announcing it had read "all 1,000 call transcripts provided." It didn't miss those 45 calls because the model is weak at search. It missed them because 45 of the 46 were in the 901 calls that were never loaded. You cannot find what you never read.

If you want the coverage ceiling in one number, it's that: same model, same key as the arms that scored 100%, landing at 2% — entirely because of how much data fit.

Figure 2. Same model, same key. The only variable that moved was how much got read.

The agent reads more — on the questions it decides to.

Every other quirk this arm showed is the same fact wearing a different hat:

• It was the most consistent arm — build-in-house asked three times gave 67% membership overlap, beating the agent's 52% and chat's 24%. Not a virtue: my packing is deterministic, so it reads the identical 99 calls every run. It's steady the way a stopped clock is steady, and consistency alone would rank it the winner.
• On enterprise pricing it guessed. "Enterprise" means ARR > $50K, a column in the warehouse. With only raw text, it inferred scale from figures mentioned on the calls and admitted several accounts "could not be confirmed."
• On sentiment it refused — a defensible call that the data type doesn't support the question, but notable that chat answered the same question over 100% of the rows. Same question, three arms, three behaviors.

Speed, decisively. These runs averaged 28 seconds — against chat's 3:40 and the agent's 11:25. One call, one answer, no loop, no queue.

And that's exactly the trap: you paste, wait twenty seconds, and a clean, confident, well-formatted answer comes back headed "I analyzed all 1,000 transcripts." Nothing in the experience tells you it's 10%.

Cost is a similar mirage — about $5.00 a question, flat, because it always fills the same window. Cheap per call; you're just paying five dollars to read a tenth of your data, with no setting for "read the rest."

Recall stays pending for every system until the answer key closes after all systems have run. Q5 recall is mechanical and final.

• Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
• Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
• Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
• Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. It closes only after all systems run - so this row stays "pending" until the final report, for every system including ours.
• Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
• Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

that I couldn't push it past about 10%, because the corpus doesn't fit and there's no setting for "fit anyway." Every failure downstream is a shadow of that one fact. The confident "all 1,000." The 1-of-46 on the one question with a mechanical answer. The 67% consistency that looks like trust and is actually a stopped clock. None of it is the model being dumb. All of it is the model being asked to answer from a tenth of the evidence and not being told which tenth.

This is the wall the rest of the series is about getting over. There are only two honest ways across it: read everything the expensive brute-force way, or pre-compute a judgment once and reuse it. Everything left in the benchmark is a variation on those two.

Part 4 — API map-reduce

My own code again, but this time built to read all 1,000 transcripts, in full, by chunking them and consolidating. Real 100% coverage, no window games. The arm that answers the question this one raises: once you finally read everything, does the answer hold still?

Methodology appendix: Anthropic Messages API, Claude Opus 4.8, 1M-token context beta (context-1m-2025-08-07) enabled, max output 8,000 tokens. Transcripts pulled from the frozen BENCHMARK_CORPUS_1K, packed newest-first to the token budget; on every run the first packed prompt exceeded context and the script backed off 15% once, landing at 99 calls / ~997K input tokens. Cost computed at Opus list ($5/$25 per M in/out). Five frozen questions, Q1 asked three times in independent calls. Full run logs (R015–R021) available.