The Coverage Benchmark, Part 1: Chat + Warehouse

If you have conversation data — sales calls, support tickets, customer check-ins, emails, internal slack conversations — you’ve likely tried to “just point Claude at them” to get insights in the last six months. Feels like a fairly obvious thing to do in 2026.

The problem is that “pointing Claude at them” could mean many different things — each with their own tradeoffs and set of decisions.

You can point to the chat app at your warehouse.

You can point an agent at it.

You can write your own code against the API — naively, or with a map-reduce pipeline, or through the Batch API, or with RAG.

You can point Claude at pre-enriched data.

You can buy a product like Rippit.

These setups all use similarly performant LLMs and they all produce confident, good-looking answers.

For each setup though there is a set of tradeoffs:

• How long does it take to get an answer?
• How accurate do I need the answer to be?
• If I ask the same question twice, do I expect to receive the same answer?
• How much does it cost?
• Does my question require the LLM to read large transcripts?

I had intuition about the tradeoffs between each of these setups, but what was completely unknown to me was the size of the tradeoffs relative to each other.

• Worse coverage - but 5% worse or 50x worse?
• Cheaper - by a buck or by $200?
• Timing - 1 minute difference or an hour difference?

I decided to do it myself, the data-driven way, and publish everything as I go - partly out of curiosity, partly because I think the results are useful to anyone sitting on a pile of conversations.

One frozen set of 1,000 of our own customer calls. Five frozen questions. Every credible approach, run under identical rules, scored on the same four axes — coverage, consistency, cost, time — one post per approach over the coming weeks.  The fifth axis, accuracy, waits for the final report — the answer key doesn't close until every system has run, so no approach (including ours) gets graded early.

And yes, at the end, Rippit takes the same exam. It will be run under the same frozen prompts and the same rules, so seven posts of results are on the record before Rippit gets a single number.

Nobody, including us, gets scored on accuracy until the answer key closes after every system has run. The final report is the full side-by-side.

This is Part 1: the setup everyone tries first.

Claude in the chat app, connected to your warehouse.

Because this is the post the whole series hangs off, here's the protocol. (Skip ahead if you just want the findings.)

The corpus. 1,000 of our own Gong sales and CS calls, frozen into a static Snowflake table on June 10.

Raw transcripts only - call ID, account, ARR, date, title, transcript.

No enrichment columns, nothing pre-computed.

~34,000 characters per call on average; ~8.5 million tokens total - about 8x what fits in the largest context window available to me.

That gap is the entire subject of this series: every system has to decide what to do about the 7.5 million tokens that don't fit.

The questions. Five, frozen verbatim, spanning a spectrum from "needs judgment" to "needs grep":

The rules.

• Fresh conversation per run.
• Prompts pasted verbatim, no follow-ups (one scripted exception below).
• Model: Claude Opus 4.8, effort high - the strongest setting, because benchmarking the weak setting would be a strawman.
• Memory paused for every run in the series.
• Every run's SQL gets extracted from the transcript and replayed, so coverage is measured, not estimated: I know exactly which calls each run touched and how many characters it actually read.

The scorecard. Four axes.

• Coverage - % of transcript characters actually read by a model.
• Consistency - same question 3x, how stable is the answer set.
• Cost - marginal $ per question (and where the spend lands - more on that below).
• Time - wall-clock, prompt to answer.
• Accuracy against a human-verified answer key comes in the final report, once every system has run (the key is built from the union of every system's candidates, so no arm gets to grade its own homework early).

Figure 1. The corpus is 8.5M tokens. The biggest context window is 1M. A typical chat run read ~120K tokens. Every system in this series is a strategy for coping with that gap.

Last study I found that chat keyword-samples your transcripts. That's still true - but it's worse than that, in a more interesting way.

I asked the identical build-in-house question three times. Same table, same model, same settings, fresh chat each time. Claude built a different analysis system every single run:

• Run 1: keyword filter → read 560-character snippets around each hit → judge from snippets. Read 0.7% of the corpus.
• Run 2: a different keyword net → pull candidate calls → read context around matches. Read 1.4%.
• Run 3: keyword net → run an LLM judge (Cortex, inside Snowflake) over the full transcripts of its 98 candidates, two passes. Read ~10%.

Across the five questions it improvised four distinct architectures - snippet sampling, candidate reading, per-row LLM classification, and a dedicated sentiment function. None of them were wrong, exactly.

The point is that the methodology your business decision rests on is selected by a die roll you never see.

Figure 2. Three runs of the identical prompt. Three different pipelines, a 14x coverage spread, and only one of them billed the warehouse.

Here's the chart that should bother you the way it bothered me.

The three runs returned 17, 17, and 25 accounts. Two identical counts! Looks stable! Except: across the three runs, 33 distinct accounts got named, and only 8 appear in all three.

Run 1 and Run 2 returned the same number and disagreed on a third of the names.

Figure 3. The counts looked stable: 17, 17, 25. The names underneath were 24% stable. Customer X sits in the found-twice bucket.

A number that swings warns you. A number that holds steady while the names underneath it change walks into your board deck unchallenged.

And the names that churn aren't the marginal ones. Customer X — the account that told us on a call they'd already moved off our product onto an internal tool — showed up in Run 1 and Run 3, and vanished in Run 2, because Run 2's hand-rolled regex happened not to catch the phrasing. Two out of three. On the single most expensive signal in the dataset, the chat-plus-warehouse architecture is a coin flip weighted by luck in keyword choice.

(Run 1, for what it's worth, found Customer X because their money quote happens to contain the literal words "internal tool." Recall by coincidence is not recall.)

Something changed since the last study, and it's the most consequential finding in this post.

When the question got hard enough - churn reasons, enterprise pricing - Claude stopped sampling and did something new: it wrote SQL that called Snowflake's in-warehouse LLM (Cortex) on every single row, classifying 1,000 transcripts per question, inside the warehouse, on our Snowflake bill. To be clear, Cortex pulling this off — running an LLM over every row, in-warehouse, on demand — is genuinely impressive infrastructure; Snowflake is a platform Rippit itself runs on. The problem isn't the engine. It's that the work is disposable.

Read that again: the chat app, which costs $0 marginal per question on a seat, quietly spent $15.80 of metered Snowflake credits on a single question.

Across the seven-question benchmark set it billed ~$29.50; counting the discarded pilot batch, benchmark day cost $75.85 of warehouse compute that the chat UI never mentioned. And it spent it again, from scratch, on the next question, because nothing persists between chats.

It also tried, in five separate runs, to CREATE TABLE — staging tables, results tables, classification scratch. Our read-only connector blocked it every time, and it worked around the block with inline queries.

Think about what it's doing: per-row LLM enrichment over 100% of the corpus, attempted persistence of the results, re-derived on every question.

It's rebuilding a conversation-intelligence pipeline, ad hoc, disposably, at query time. The model already knows what the right architecture is. It just can't keep it.

Figure 4. Source: Snowflake AI_FUNCTIONS metering, token-share allocation. The $0 bars read ~1% of the data; the tall bars rebuilt per-row classification and threw it away.

Last study's ugliest result was the sentiment question: a polished, confident, monthly trend chart that was statistically noise.

This time, when I asked the same question, Claude refused. Thirty-eight seconds in Claude states:

I read 3 transcripts out of 1,000 - roughly 0.3%... Anyone who tells you a confident monthly sentiment trend off this table either silently ran an LLM over the transcripts (and should say so) or is making it up.

That's a real improvement, and worth stating plainly: the model's honesty at high effort is now excellent. When I told it to proceed, it ran a sentiment model over all 1,000 calls, reported a flat trend, separated "what a model scored" (100%) from "what I actually read" (under 1%), and flagged - unprompted - that scores skew positive because the rep's voice is mixed into every transcript. It even caught a real skew in my benchmark corpus's date distribution that I'd missed myself.

The asterisk: honesty about sampling doesn't fix sampling. The refusal is the system telling you, in its own words, that the architecture it's been handed can't answer the question.

In five of seven runs, unprompted, Claude reached for pre-computed enrichment:

• Twice it tried to load our product's own analysis tools mid-run (it found them in the connector list), reasoned about using them, and backed off only because the benchmark table isn't in that workspace. Its words: per-row enrichment is "exactly what this is built for."
• On sentiment, its recommended next step was the role-aware, pre-labeled sentiment column - the one that exists in our enriched tables.
• On churn, it cross-referenced its answer against "the verified churn drivers from your enriched data" - a table it never read but knew it wanted.

Last study I called this "the tell" and it happened twice. At benchmark scale it's five out of seven. The system under test keeps independently specifying the system it's being tested against.

Two of the five questions, the raw setup performed well, and the final scorecard will say so:

• The keyword control (Q5): one SQL filter, 46 calls found, 28 seconds, $0. If your question is "which calls mention Competitor X," you do not need a vendor. Grep wins. It will win in the final table too.
• Enterprise pricing (Q4): because "enterprise" lives in a structured ARR column, Claude filtered to the right 416 calls by definition, classified each one, and produced a genuinely useful five-theme answer for $5.30 metered. When the segment is a column and not a judgment, the raw setup is respectable.

Figure 5. The same spectrum as study 2, confirmed under benchmark conditions. The questions that drive renewals and roadmaps live on the right side.

The pattern from study 2 holds at benchmark scale: performance is a function of how keyword-shaped your question is. The questions that drive renewals, roadmaps, and board decks are the least keyword-shaped questions you have.

Every column including our own stays empty until that system runs under the same frozen rules.

¹ The 0.7–41% range covers reasoning-reads - characters an LLM read and reasoned over to build the answer. Q3's run additionally pushed 100% of transcripts through a scalar sentiment-scoring model after disclosing it had read 0.3%; see Finding 4.

² Averages are unweighted means over the seven scored runs (Q1 counts three times; the Q5 control's 0% LLM-read is included - excluding it, average coverage is 11.1%).

• Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
• Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
• Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
• Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. It closes only after all systems run - so this row stays "pending" until the final report, for every system including ours.
• Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
• Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

Part 2 - the agentic surfaces.

Cowork and Claude Code, same corpus, same questions, same rules. The pitch for agents is that they loop, write code, and check their work. I'll find out what that buys when the data is 8x the context window.

Methodology appendix: corpus freeze date June 10, 2026; run dates June 11; all SQL extracted and replayed for coverage measurement; full run log available. Disclosure: an initial five-run batch was discarded when I found cross-run contamination - Claude's memory feature was linking my "fresh" sessions. All runs reported above were re-executed with memory paused.