The Coverage Benchmark, Part 2: The Agent
Mike Nucci·July 28, 2026What is the Coverage Benchmark?
Everyone with conversation data is trying to get insights out of it with Claude, and there are a lot of ways to do that: chat on a warehouse, agents, custom code, RAG, a purpose-built product. There are tradeoffs between them and we wanted to understand the scale. Is one approach 5% worse than another, or 50x worse? Cheaper by a dollar, or by $200? A minute slower, or an hour?
So we built a benchmark. Same 1,000 frozen customer calls, the same five questions, for every approach, scored on coverage, consistency, cost, and time.
Series note: Part 2 of the Coverage Benchmark. Part 1 tested Claude in the chat app connected to your warehouse. Same 1,000 frozen calls, same five frozen questions — this time the agent.
Result in brief
- Coverage depended on the question: about 95% on churn reasons, where it fanned out about 25 sub-agents, and about 1% on build-in-house. It averaged 31%, against chat’s 10%.
- Consistency: 22, 25 and 25 accounts; 31 named, 16 in all three, 52% overlap against chat’s 24%. Customer X appeared in all three runs.
- Time: 11 minutes a question on average, against chat’s 3:40. The near-full read of Q2 took 31:44, and two sub-agents failed mid-run.
- Cost: $0 to the warehouse; every read ran on the Claude seat.
- Where it works: the keyword control (Q5) found all 46 calls.
Our reading: steadier counts are still not a stable result. Half the named accounts changed between identical runs.
The pitch for agents
Part 1 pointed the chat app at the warehouse and watched it keyword-sample your transcripts.
The obvious counterargument — the one I believed myself — is that chat is the weak setup.
I assumed a real coding agent would write code, run it, check the result, loop and verify its own work.
An agent shouldn't sample 1% of your data and call it done. It should build an actual pipeline.
So I ran the same benchmark on Claude Code — same Claude Opus 4.8 at high effort, connected to the same frozen Snowflake table, memory off.
Five frozen questions, Q1 asked three times. I went in expecting higher coverage.
I was kind of right, but still wrong enough that I wouldn’t give my boss these answers. It turned out to be more interesting than a clean win or loss.
What the agent actually did, question by question
Here are the results:

Not one query called the warehouse's LLM.
Hold onto that — it's the whole cost story, and it's the opposite of what chat did.
Finding 1: on the hard question, the agent finally built the pipeline
On Q2 — "top churn reasons, ranked, with counts" — the agent did exactly what we’d all think it would do.
It pulled transcripts out to files, then spawned about 25 sub-agents - each reading a batch of full transcripts and classifying them, then consolidated. That's a real map-reduce, and it read something like 95% of the corpus - not the 14K-character truncation chat used on the same question in Part 1.
So yes: the agent read more. On this question it read almost everything.

The agent reads more — on the questions it decides to.
Finding 2: it cost 32 minutes and left a mess
Here's the other half. That Q2 run took 31 minutes and 44 seconds. Chat answered the same question in eight.
Just a reminder — this is only 1,000 calls, a small corpus; many teams generate that many in a day or two.
Two of the sub-agents died mid-response with API errors and had to be re-run. When it finished, it left a shell still running in the background that I had to clean up by hand.This is the agent tax nobody demos. The loop that lets it read everything is the same loop that runs long, fails partway, retries, and leaves orphans.
Across all of the runs the agent averaged 11 minutes a question against chat's 3:40 - and the single longest run was the one that did the work I actually asked it to do. Reading everything the hard way is slow and brittle.

Finding 3: it’s still a roll of the dice
Part 1's headline was that chat picks a different analysis architecture every run. The agent does too. It's just different dice.
- Rare-event question (Q1)? Keyword sampling, ~1% — the exact same shortcut chat took. Three runs, three keyword nets, none of them a pipeline. The agent that fanned out 25 sub-agents for churn used a quick keyword search for build-in-house.
- Aggregate question (Q2)? The 25-sub-agent map-reduce described above.
- Structured-key question (Q4)? Because "enterprise" is an ARR > $50,000 column, it filtered by definition to 93 accounts and read their pricing sentences - cheap and clean, same as chat. When the segment is a column, it works with ease.
- Keyword control (Q5)? One scan, found all 46 Competitor-X calls — 100% recall, matches the answer key exactly. Then it spent five more minutes building a competitive battle card I never asked for. Chat did the identical 46 in 28 seconds and stopped. Same right answer, about 12× the time.
The methodology still isn't chosen by the data.
It's chosen by a dice roll you don't see.
Finding 4: same question, same agent — still different customers
The three build-in-house runs returned 22, 25, and 25 accounts. Across them, 31 distinct accounts got named, and 16 appear in all three — 52% overlap.
That's genuinely better than chat's 24%. The agent is more consistent — Part 1 predicted this, and it held: the looping and self-checking do stabilize the answer set. Customer X, the account that already moved to an internal tool, showed up in all three agent runs, where chat found it only two out of three times.
But read that number the right way. 52% consistency means about half the named accounts change when you ask the identical question again.
You'd still walk away with a materially different customer list depending on which run you happened to keep.

The agent is steadier. Steadier is not stable.
Where the agent wins, honestly
- Keyword control (Q5): 46/46, matches the key. Raw setups own this row - though the agent gave back its speed advantage by over-analyzing.
- Consistency: 52% beats chat's 24%. If you must use a sampling setup, an agent gives you a steadier sample.
- Your warehouse bill: $0. Every read ran on the Claude seat, not Cortex. Where chat quietly billed ~$29.50 of warehouse compute across seven scored runs (five questions, Q1 three times), the agent billed nothing to Snowflake. If you're watching a Cortex line item, the agent is the cheaper place to spend — you pay in wall-clock and fragility instead.
The scorecard so far

How to read the scorecard (this key appears in every part)
- Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
- Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
- Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
- Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. Scoring against it is a separate evaluation, not part of these results, so this row stays "pending" for every system, including ours.
- Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
- Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.
The takeaway
I expected the agent to beat chat on coverage. It did — sometimes. On the aggregate question it read nearly everything; on the rare-event question it read 1%, same as chat. Averaged out, it reads more (31% vs 10%) and repeats itself more (52% vs 24%) — real improvements — but it buys them with 3× the wall-clock, sub-agents that die mid-run, and a habit of reaching for pre-computed data the moment you stop watching.
The agent is a better sampler. It is not a system that reliably reads your data, answers the same way twice, or does it fast. The gap between "an agent that can loop" and "a system you'd trust with a churn number in a board deck" is still the whole thing — and, once again, the agent itself kept trying to close that gap by reaching for pre-computation.
What this means for your team
An agent is a good investigator for a one-off question. Turning it into a recurring workflow is a different job. Coverage swung from about 95% to about 1% depending on the question, the near-full read took 31:44 with two sub-agents failing mid-run, and only half the named accounts held between identical runs. A weekly report built on that would change for reasons nobody on the team chose. For workflows that run on a schedule, the read has to cover every call the same way each time, with each verdict saved where both the agent and the team can query it. The series recap compares all eight setups by the job each one suits.
Next up
Part 3 — the naive API
No agent, no chat, no warehouse LLM — just my own code stuffing as many transcripts as fit into one big call. The first arm where I control exactly how much gets read, and the first hard ceiling of the series.
Methodology appendix: Claude Code, Opus 4.8 high effort, memory off, one fresh session per run, all runs against the frozen BENCHMARK_CORPUS_1K. From Q3 onward the prompt was pinned to the corpus table only, after an unpinned run silently joined a pre-computed enrichment table; that run is logged and excluded. Coverage and architecture verified from Claude Code's own session logs (JSONL), not terminal capture. Full run log available.
Keep reading
More from the Rippit Library.
Same Count, Different Customers
Claude on a warehouse read 0.7–41% of the calls, built a different method each run, and quietly billed $15.80 of Cortex credits on one question.
Half the Price, the Same Unstable List
Moving the same full read to the Batch API cut the cost by more than half; agreement across runs stayed at 29%.
What It Takes to Trust an AI Answer About Your Customers
The series in one page: which approach read everything, which gave the same list twice, and what each means for a board number or a renewal list.