The Coverage Benchmark, Part 7: Claude + Pre-Computed Enrichment

Part 7 of the Coverage Benchmark. Every arm so far answered by reading transcripts - a little, all of them, or a chosen few. This one doesn't read transcripts at all. It reads judgments someone already made about every call and stored as columns.

The first test that doesn't read the transcripts

Every test so far was unique in how I orchestrated Claude reading the transcripts.

But no matter how I sliced it, you have to re-derive the answer from raw text at question time, and that's where it breaks. So this test  stops reading raw text.

Instead it uses pre-computed enrichment: for every one of the 1,000 calls (I’ve already prompted Claude for various use cases and had the results saved as meta-data on the calls) sentiment, churn risk, competitors mentioned, topics, a summary, key words, the opportunity amount

This time, I hand Claude that, the whole enrichment layer for all 1,000 calls, and ask the same five frozen questions. No transcripts. Just the persisted analysis.

This is the architecture every earlier test kept accidentally reaching for.

Here's why this changes everything.

A raw transcript averages ~34,000 characters; the full corpus is ~8.5M tokens, which is why nothing could read it all in one shot.

But the enrichment of a call is a few hundred characters. Enriched, the entire 1,000-call corpus is about 737K tokens. It fits in a single context window.

So Claude reads 100% of the corpus's pre-enriched meta-data in one call - no sampling, no map-reduce, no retrieval gamble - for $3.72, in 37 seconds.

Figure 1. Persistence compresses each call to a verdict, shrinking the corpus from 8.5M tokens (doesn't fit) to 0.74M (one call). The coverage problem was a size problem all along.

*100% of the persisted judgments - Claude reads the enrichment of every call, not the raw text. This is a reasoning-read over analysis, footnoted separately from raw-transcript reads.

The expensive part - reading every full conversation - already happened once, upstream in DBT, when I built the enrichment pipelines.

Every question after that is a cheap query against the result. That's the entire premise the series has been walking toward: read once, reuse forever.

On the build-in-house question, reading only the enrichment, Claude returned 12 accounts, every one quote-anchored.

It caught Customer X, the account that already moved to an internal tool, which the naive read missed entirely.

Recall that RAG - clever retrieval, same model - found zero on this exact question. The difference isn't the reasoning; it's that the build-vs-buy signal survived into the enrichment summaries, where retrieval couldn't surface it from raw text.

And Claude did a great job: it read enrichment mentioning "build" and still excluded the ones that meant building a dashboard inside the product. High precision, real accounts, from a layer a hundredth the size of the corpus.

Asked three times, the build-in-house list came back 12, 13, 13 accounts, with 10 of 15 appearing in all three runs: 67% consistency. That's the steadiest result of any test where Claude does the reasoning: triple the live map-reduce's 22%.

Figure 2. The same 67%, opposite meanings: the naive call is steady because it reads the same blind 10% every run; C1 is steady because it reads 100% of the (frozen) judgments. C1 is the first arm that's both complete and stable.

The reason it's steady is the reason it matters: the input is frozen. Claude reads the identical enrichment every run, so it lands on nearly the same list. Persistence buys stability that re-derivation never could.

C1 is the best setup yet. It is not the finish line, and the two places it falls short are precise.

1. Recall is capped by what the enrichment preserves. On the Competitor-X control, Claude found 23 of 46 calls - 50%. It mined the enriched text well , but the generic summaries had already dropped half the mentions before Claude ever saw them. You inherit the enrichment's fidelity, and this enrichment was built for general analysis, not for this question. It kept what it was designed to keep.

2. 67% still isn't 100% - because C1 froze the reading, not the verdict. The enrichment is fixed, but "does this account count as considering build-in-house?" is still a judgment Claude re-makes over that enrichment on every run. So a third of the list still wobbles. The reading is persisted; the verdict is not.

Both cracks have the same shape: the persisted layer is generic, and the final answer is still re-derived from it at question time.

The fix isn't a better generic enrichment.

It's persisting the judgment for the actual question, and doing it on demand.

Instead of hoping a general-purpose summary happened to keep the build-in-house signal, you ask the question, classify all 1,000 calls for that question against the full transcripts, and store the verdict (yes/no) with the quote as a column.

Then recall isn't capped by a generic summary and the answer doesn't change.

That is the one move no test in this series has made yet. It's the last test.

*C1 coverage is 100% of the persisted judgments - a reasoning-read over pre-computed enrichment, not a raw-transcript read. Q1 recall stays pending until the key closes after all systems run; C1 named ~12 accounts at high precision.

• Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
• Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
• Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
• Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. It closes only after all systems run - so this row stays "pending" until the final report, for every system including ours.
• Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
• Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

Pre-computed enrichment is the first test that gets the shape of the answer right: 100% coverage, a few dollars, half a minute, and the steadiest Claude-reasoned result in the benchmark - because the reading was done once and stored, so the corpus finally fits and the answer finally mostly holds. It found build-in-house accounts that clever retrieval scored zero on.

Its two limits are not failures of the idea; they're the boundary of the generic version of it.

1. Recall is capped by what a general-purpose summary happened to keep;
2. And the last-mile verdict is still re-derived, so it still wobbles a third of the time.

Both dissolve the moment the persistence is aimed at the specific question and made against the full text.

Part 8 — Rippit. The finale.

The arm that does exactly that: ask a question, classify every one of the 1,000 conversations against the full transcript for that question, and persist the verdict as a column. Same corpus, same five frozen questions, same scoring - graded under identical rules to everything above. This is the one the whole benchmark was built to test, and it runs last on purpose, so nothing about it led the witness.

Methodology appendix: Claude Opus 4.8, 1M-token context beta enabled, one call per question over the pre-computed enrichment for all 1,000 calls (columns: sentiment, churn risk, competitors mentioned, topics, contact reason, product areas, key words, summary, opportunity amount, issue resolved, urgency; ~737K input tokens). No raw transcripts in context. Cost at Opus list ($5/$25 per M in/out); the enrichment build is a prior, sunk cost, noted but not counted as marginal. Five frozen questions, Q1 asked three times. Coverage reported as 100% of persisted judgments and footnoted as such. Full run logs (R042–R048) available.