The Coverage Benchmark, Part 5: The Batch API

Part 5 of the Coverage Benchmark. This is the shortest post in the series on purpose. Part 4 read all 1,000 transcripts with a live map-reduce. This is the exact same code, submitted as a batch job - so it's really a test of one thing: does buying the same answer for half the price change the answer?

The same machine, at half price

Part 4 was the first time I forced Claude to actually read 100% of the 1,000 transcripts — chunk them up, read every word, stitch the findings back together.

It got to full coverage, and it cost about $61 a question.

It also had another problem: asked the same question three times, it handed back three different customer lists — only 22% overlap.

This run changes nothing about how Claude reads or analyzes the calls. Same model, same chunks, same prompts, same everything.

The one difference is delivery: instead of making the calls live, I hand them to Anthropic's Batch API. If I'm more concerned about price than time, then this would be the way to go vs. the prior run.

Price is the obvious knock on Part 4, and batch is the obvious fix. Relative to Part 4, the batch API holds everything constant except cost and time, so I can find out whether that 22% consistency was a fluke - or something else.

The two things batch is supposed to move, moved: cost fell by more than half, and time stretched from two minutes to an asynchronous window.

The two things that measure whether you can trust the output, consistency and control recall, landed right where Part 4 left them. Both arms read everything; but neither did well on Consistency or Recall.

Figure 1. The two things batch is designed to move (left) moved; the thing that measures trust (right) didn't.

Part 4's uncomfortable finding was that reading 100% of the corpus three times produced three different customer lists - 22% overlap.

The easy hope is that it was an artifact of how I ran it: live calls, concurrency, some timing quirk.

Unfortunately the batch run isn’t any different.

Run the identical method through completely different infrastructure — async queue, server-side scheduling, results retrieved hours later - and the inconsistency is still there: 29% overlap, statistically the same. The 7-point difference between 22% and 29% isn't batch being steadier; it's the gap you'd expect between any two runs of an identically noisy process.

So the conclusion from putting map-reduce and batch side by side is that the instability is intrinsic to re-deriving the judgment, not to the machinery you re-derive it on.

You cannot buy your way out of it with cheaper compute. Half price on an answer you can't reproduce is still an answer you can't reproduce.

Moolah! 💸💸💸💸💸

And that's not nothing when we’re talking 10’s of dollars per question answered. If you're going to run map-reduce anyway and you don't need the answer in the next two minutes, batch is the right choice: the identical output for less than half the money.

Of the two brute-force arms, this is the rational one to actually turn on in production.

It's just that "the rational way to do it" and "a way you'd trust it" are two different things.

Batch is the efficient way to buy the same unstable answer.

Recall on build-in-house stays pending for every system until the key closes after all systems run. Q5 recall is a text scan for the sampling arms and an LLM full-read for map-reduce/batch — the difference is itself a finding.

• Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
• Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
• Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
• Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. It closes only after all systems run - so this row stays "pending" until the final report, for every system including ours.
• Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
• Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

Batch did precisely what it advertises: the same 100% read for less than half the price, traded against latency. And that's the point of running it. By holding the method constant and moving only cost and time, it proves that Part 4's instability — the 22% that's now 29% — lives in the approach, not the infrastructure. Reading everything doesn't stabilize a judgment whether you read it live or overnight, at full price or half. The defect is that the judgment is thrown away and rebuilt every time, and no billing model fixes that.

Which finally closes the "just read everything" branch of this series.

Two arms, live and batched, both at 100% coverage, both unreliable and both expensive given we’re getting the wrong answers.

Part 6 — RAG

The first arm that stops trying to read everything and instead tries to be smart about what to read — retrieve only the passages that look relevant, then reason over those. It's a genuinely different bet than Parts 3–5, and it comes with a genuinely different failure mode: you can only reason over what retrieval hands you.

Methodology appendix: Anthropic Message Batches API, Claude Opus 4.8, identical map-reduce to Part 4 (50 batches of 20 transcripts, full read, one reduce). Batch pricing is 50% of standard list ($2.50/$12.50 per M in/out for the map phase; the single reduce call is a standard live call). One run (Q1, R029) had 2 of 50 map batches error server-side and is scored at 96% coverage, not 100%. Batch turnaround, collected promptly, was ~4–42 min; four runs were collected ~3 days after submission and their submit-to-collect wall clock is discarded as a collection artifact. Five frozen questions, Q1 asked three times. Full run logs (R028–R034) available.