The Coverage Benchmark, Part 6: RAG

Part 6 of the Coverage Benchmark. Parts 3–5 all tried to answer by reading - a little (naive), everything (map-reduce), everything again for less (batch). This arm makes the opposite bet: don't read more, read smarter. Retrieve only the passages that look relevant, and reason over those.

The opposite bet

Every test so far competed on how much of the actual transcripts the LLM could read and analyze.

RAG objects the premise and instead of reading more - it reads selectively.

1. Chunk all 1,000 transcripts into smaller passages.
2. Embed them into a vector index
3. For each question select only the handful of passages most similar to the question.
4. Then answer from those passages.

I built exactly that - 16,546 passages, a local embedding model, cosine search, top-50 passages per question fed to the same Claude Opus 4.8 every other test used.

This is the test that's supposed to be clever: why read eight million tokens when the answer lives in ~50 chunks?

It is the cheapest, fastest test in the entire benchmark.

Unfortunately - turns out it’s a complete waste of time.

I’d rather sweat on the subway platform in July than rely on this approach.

Coverage is ~0.35% of the corpus text - a whopping ⅓ of 1 percent!

the lowest of any test by a massive margin, because RAG reads snippets - it never reads a whole call.

Each question cost about $0.25 and ran in ~19 seconds.

Figure 1. RAG costs a fraction of any other arm - and it's the only one that answered the build-in-house question with zero. Cheapest, fastest, and least accurate are the same fact.

I expected RAG to run away with Q5 - "which calls mention Competitor X."

A named competitor is a needle; surely similarity search finds needles.

Well, ya know what? I was wrong.

Retrieval surfaced 13 of the 46 calls that mention them. A plain text scan - what the chat and agent arms effectively ran - finds all 46 for free.

Semantic search doesn't, because "Competitor X" is a rare proper noun, and a dense embedding of "which calls mention the competitor" drifts toward competitor talk in general rather than that one exact token.

RAG lost to grep.

Figure 2. The one question with a mechanical answer, across every arm. RAG's clever retrieval lands below both brute-force reads and far below a plain text scan.

The failure isn't a weak embedder or a cheap budget - it's structural, and worth stating explicitly because it's the reason I wanted to do the back half of this series.

"Find every account that considered building in-house" is not a similarity query.

The answer isn't a cluster of passages that resemble the question; it's a verdict scattered across the whole corpus, phrased a hundred different ways ("we'd probably just do it ourselves," "the data team can spin that up," "engineering wants a crack at it") - almost none of which resemble the sentence "considering building an in-house alternative."

Top-K similarity retrieves the passages nearest the question, which is the wrong target when the answer is stated very differently.

You cannot retrieve your way to a complete list of something that requires context instead of specific words.

*RAG named zero accounts on Q1, so its recall is 0% regardless of the final key. Every other arm's Q1 recall stays pending until the key closes after all systems run. Q5 recall is a text scan for the sampling arms and an LLM read for the map-reduce/RAG arms.

• Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
• Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
• Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
• Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. It closes only after all systems run - so this row stays "pending" until the final report, for every system including ours.
• Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
• Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

RAG is the most efficient test in this benchmark: a third of a percent of the text, a quarter a question, nineteen seconds.

But it finished last on accuracy - zero on build-in-house, 28% on just finding a competitor’s name.

Not because it read too little, but because similarity is the wrong RAG target for a "find every X" question. The answer to those questions doesn't look like the question, so no amount of cleverness at the retrieval step surfaces it.

That closes the loop this series has been walking:

Four arms, three strategies, one shared defect - every one of them re-derives the judgment from the raw text at question time, and re-derivation is either partial, unstable, or misdirected.

There is only one move left that none of them make: stop re-deriving.

Read every conversation once, make the judgment once - is this account considering building in-house? yes/no, with the quote - and store it as a column.

Then "find every account that…" is a lookup that's complete by construction, instant, and identical every time you run it.

Part 7 — Claude + pre-computed enrichment

This will be the first arm that doesn't answer from raw transcripts at all, but from judgments already made and stored as meta data on the transcript. It's the architecture every arm so far kept accidentally reaching for - and the one this whole benchmark has been walking toward.

Methodology appendix: 1,000 transcripts chunked into 16,546 overlapping ~2,300-char passages, embedded locally with BAAI/bge-small-en-v1.5 (384-dim, normalized cosine), index built once and cached. Retrieval budget top-50 passages per question (the "realistic" setting). Answers generated by Claude Opus 4.8 over the retrieved passages only, max 8,000 output tokens; cost at Opus list ($5/$25 per M in/out). Coverage = retrieved passage characters / total corpus characters. Five frozen questions, Q1 asked three times; retrieval is deterministic, so the Q1 context was identical across runs. Full run logs (R035–R041) available.