The Coverage Benchmark, Part 5: The Batch API
Mike Nucci·August 19, 2026What is the Coverage Benchmark?
Everyone with conversation data is trying to get insights out of it with Claude, and there are a lot of ways to do that: chat on a warehouse, agents, custom code, RAG, a purpose-built product. There are tradeoffs between them and we wanted to understand the scale. Is one approach 5% worse than another, or 50x worse? Cheaper by a dollar, or by $200? A minute slower, or an hour?
So we built a benchmark. Same 1,000 frozen customer calls, the same five questions, for every approach, scored on coverage, consistency, cost, and time.
Part 5 of the Coverage Benchmark. This is the shortest post in the series on purpose. Part 4 read all 1,000 transcripts with a live map-reduce. This is the exact same code, submitted as a batch job - so it's really a test of one thing: does buying the same answer for half the price change the answer?
Result in brief
- Same map-reduce as Part 4, submitted through Anthropic’s Batch API: cost fell from about $61 to about $28 a question (54% less).
- Time went from about two minutes to an asynchronous window, about 4 to 42 minutes when collected promptly.
- Coverage: 100% (one run 96%, after 2 of 50 map batches errored).
- Consistency: 29% overlap, against 22% live. Control recall (Q5): 54%, against 78% live.
Our reading: cheaper compute changed the bill, not the list. The instability comes from re-deriving the judgment, not from the infrastructure.
The same machine, at half price
Part 4 was the first time I forced Claude to actually read 100% of the 1,000 transcripts — chunk them up, read every word, stitch the findings back together.
It got to full coverage, and it cost about $61 a question.
It also had another problem: asked the same question three times, it handed back three different customer lists — only 22% overlap.
This run changes nothing about how Claude reads or analyzes the calls. Same model, same chunks, same prompts, same everything.
The one difference is delivery: instead of making the calls live, I hand them to Anthropic's Batch API.
Price is the obvious knock on Part 4, and batch is the obvious fix. Batch holds everything constant except cost and time, so it tests whether Part 4's 22% consistency was a fluke.
What changed, and what didn't

The two things batch is supposed to move, moved: cost fell by more than half, and time stretched from two minutes to an asynchronous window.
The two things that measure whether you can trust the output, consistency and control recall, landed right where Part 4 left them. Both arms read everything, but neither was consistent or recalled well.

The instability was never the plumbing
Part 4's uncomfortable finding was that reading 100% of the corpus three times produced three different customer lists - 22% overlap.
The easy hope is that it was an artifact of how I ran it: live calls, concurrency, some timing quirk.
Unfortunately the batch run isn’t any different.
Run the identical method through completely different infrastructure — async queue, server-side scheduling, results retrieved asynchronously - and the inconsistency is still there: 29% overlap, just as unstable. The 7-point difference between 22% and 29% isn't evidence that batch is steadier; with three runs each, both lists churned about as much.
So the conclusion from putting map-reduce and batch side by side is that the instability is intrinsic to re-deriving the judgment, not to the machinery you re-derive it on.
You cannot buy your way out of it with cheaper compute. Half price on an answer you can't reproduce is still an answer you can't reproduce.
Where batch wins
At tens of dollars per question, halving the bill matters. If you're going to run map-reduce anyway and you don't need the answer in the next two minutes, batch is the right choice: the identical output for less than half the money.
Of the two brute-force arms, this is the rational one to actually turn on in production.
It's just that "the rational way to do it" and "a way you'd trust it" are two different things.
Batch is the efficient way to buy the same unstable answer.
The scorecard so far

Recall on build-in-house stays pending for every system; it’s scored in a separate evaluation, not part of these results. Q5 recall is a text scan for the sampling arms and an LLM full-read for map-reduce/batch — the difference is itself a finding.
How to read the scorecard (this key appears in every part)
- Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
- Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
- Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
- Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. Scoring against it is a separate evaluation, not part of these results, so this row stays "pending" for every system, including ours.
- Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
- Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.
The takeaway
Batch did precisely what it advertises: the same 100% read for less than half the price, traded against latency. And that's the point of running it. By holding the method constant and moving only cost and time, it shows that Part 4's instability — the 22% that's now 29% — lives in the approach, not the infrastructure. Reading everything doesn't stabilize a judgment whether you read it live or as a batch, at full price or half. The defect is that the judgment is thrown away and rebuilt every time, and no billing model fixes that.
Which finally closes the "just read everything" branch of this series.
What this means for your team
If budget is your constraint, batch is the right lever: the same full read for less than half the price, about $28 a question instead of about $61. It doesn’t change what the money buys. At 29% overlap, a recurring analysis run through batch still returns a different set of customers each time, and the wait moves from about two minutes to an asynchronous window of roughly 4 to 42 minutes. For an analysis your team runs every week or month, the larger lever, as Parts 7 and 8 show, is not re-deriving the judgment each time: store each verdict once and the next run starts from it. The series recap shows where batch fits among the eight approaches.
Next up
Part 6 — RAG
The first arm that stops trying to read everything and instead tries to be smart about what to read — retrieve only the passages that look relevant, then reason over those. It's a genuinely different bet than Parts 3–5, and it comes with a genuinely different failure mode: you can only reason over what retrieval hands you.
Methodology appendix: Anthropic Message Batches API, Claude Opus 4.8, identical map-reduce to Part 4 (50 batches of 20 transcripts, full read, one reduce). Batch pricing is 50% of standard list ($2.50/$12.50 per M in/out for the map phase; the single reduce call is a standard live call). One run (Q1, R029) had 2 of 50 map batches error server-side and is scored at 96% coverage, not 100%. Batch turnaround, collected promptly, was ~4–42 min; four runs were collected ~3 days after submission and their submit-to-collect wall clock is discarded as a collection artifact. Five frozen questions, Q1 asked three times. Full run logs (R028–R034) available.
Keep reading
More from the Rippit Library.
Reading Everything Isn’t Enough
Map-reduce read every call for about $61 a question, yet three runs named 73 accounts and agreed on only 16.
The Agent Reads More, but Not the Same Way Twice
Claude Code read 95% on one question and 1% on another, averaged 11 minutes, and kept only half its account list on a re-ask.
Same Count, Different Customers
Claude on a warehouse read 0.7–41% of the calls, built a different method each run, and quietly billed $15.80 of Cortex credits on one question.