The Coverage Benchmark, Part 2: The Agent

Series note: Part 2 of the Coverage Benchmark. Part 1 tested Claude in the chat app connected to your warehouse. Same 1,000 frozen calls, same five frozen questions — this time the agent.

The pitch for agents

Part 1 pointed the chat app at the warehouse and watched it keyword-sample your transcripts.

The obvious counterargument — the one I believed myself — is that chat is the weak setup.

I assumed that if you gave the model a real coding agent: it would write code, run it, see the result, loop, and check its own work.

An agent shouldn't sample 1% of your data and call it done. It should build an actual pipeline.

So I ran the same benchmark on Claude Code — same Claude Opus 4.8 at high effort, connected to the same frozen Snowflake table, memory off.

Five frozen questions, Q1 asked three times. I went in expecting higher coverage.

I was kind of right, but still wrong enough that I wouldn’t give my boss these answers. It turned out to be more interesting than a clean win or loss.

Here are the results:

Not one query called the warehouse's LLM.

Hold onto that — it's the whole cost story, and it's the opposite of what chat did.

On Q2 — "top churn reasons, ranked, with counts" — the agent did exactly what we’d all think it would do.

It pulled transcripts out to files, then spawned about 25 sub-agents - each reading a batch of full transcripts and classifying them, then consolidated. That's a real map-reduce, and it read something like 95% of the corpus - not the 14K-character truncation chat used on the same question in Part 1.

So yes: the agent read more. On this question it read almost everything.

Figure 1. Coverage = % of the 1,000-call corpus read per question, chat (A1) vs. agent (A2). Q3 A1 is a scalar sentiment scan, not a reasoning-read; Q4 shown corpus-wide; Q5 is a mechanical keyword scan, 0% LLM-read.

The agent reads more — on the questions it decides to.

Here's the other half. That Q2 run took 31 minutes and 44 seconds. Chat answered the same question in eight.

Just a reminder — this is only 1,000 calls. Any company could have 1,000 support calls in half a day.

Two of the sub-agents died mid-response with API errors and had to be re-run. When it finished, it left a shell still running in the background that I had to clean up by hand.This is the agent tax nobody demos. The loop that lets it read everything is the same loop that runs long, fails partway, retries, and leaves orphans.

Across all of the runs the agent averaged 11 minutes a question against chat's 3:40 - and the single longest run was the one that did the work I actually asked it to do. Reading everything the hard way is slow and brittle.

Figure 2. Each point is one question run. Y = % of the 1,000-call corpus read; X = wall-clock minutes per question. Shaded band is chat's typical range (under 4:40). The 95%-coverage run took 32 minutes and lost two sub-agents mid-response.

Part 1's headline was that chat picks a different analysis architecture every run. The agent does too. It's just different dice.

• Rare-event question (Q1)? Keyword sampling, ~1% — the exact same shortcut chat took. Three runs, three keyword nets, none of them a pipeline. The agent that fanned out 25 sub-agents for churn had his feet on his desk when asked about building-in-house.
• Aggregate question (Q2)? The 25-sub-agent detailed above.
• Structured-key question (Q4)? Because "enterprise" is an ARR > $50,000 column, it filtered by definition to 93 accounts and read their pricing sentences - cheap and clean, same as chat. When the segment is a column, it works with ease.
• Keyword control (Q5)? One scan, found all 46 Competitor-X calls — 100% recall, matches the answer key exactly. Then it spent five more minutes and credits that I paid for - building a competitive battle card I never asked for. Chat did the identical 46 in 28 seconds and stopped. Same right answer, 11× the time.

The methodology still isn't chosen by the data.

It's chosen by a dice roll you don't see.

The pit boss just walked over because the agent just rolled the dice off the craps table, from 1% to 95%.

The three build-in-house runs returned 22, 25, and 25 accounts. Across them, 31 distinct accounts got named, and 16 appear in all three — 52% overlap.

That's genuinely better than chat's 24%. The agent is more consistent — study 2 predicted this, and it held: the looping and self-checking do stabilize the answer set. Customer X, the account that already moved to an internal tool, showed up in all three agent runs, where chat found it only two out of three times.

Queue Meatloaf — cause “Two out of Three Ain’t Bad”.

But read that number the right way. 52% consistency means half the named accounts still change when you ask the identical question a second time.

Barely better than the odds at the roulette table doesn’t feel great when already out a couple hundred.

You'd still walk away with a materially different customer list depending on which run you happened to keep.

Figure 3. Share of named accounts appearing in all three runs of the identical question, chat (A1) vs. agent (A2). Dotted line at 100% marks full run-to-run consistency.

The agent is steadier. Steadier is not stable.

• Keyword control (Q5): 46/46, matches the key. Raw setups own this row - though the agent gave back its speed advantage by over-analyzing.
• Consistency: 52% beats chat's 24%. If you must use a sampling setup, an agent gives you a steadier sample.
• Your warehouse bill: $0. Every read ran on the Claude seat, not Cortex. Where chat quietly billed ~$29.50 of warehouse compute across the question set, the agent billed nothing to Snowflake. If you're watching a Cortex line item, the agent is the cheaper place to spend — you pay in wall-clock and fragility instead.

Recall stays pending for every system until the answer key closes after all systems have run.

¹ The 0.7–41% range covers reasoning-reads - characters an LLM read and reasoned over to build the answer. Q3's run additionally pushed 100% of transcripts through a scalar sentiment-scoring model after disclosing it had read 0.3%; see Finding 4.

² Averages are unweighted means over the seven scored runs (Q1 counts three times; the Q5 control's 0% LLM-read is included - excluding it, average coverage is 11.1%).

• Coverage - % of the corpus's transcript characters an LLM actually read to produce the answer. Measured, not estimated: I extract every query each run issued and replay it, so partial reads (snippets, truncations) count fractionally. Scalar scoring passes (e.g. a sentiment function) are footnoted separately from reasoning-reads.
• Consistency - the same question asked three times in fresh sessions; the score is the % of named accounts that appear in all three answers. Stable-looking totals can hide membership churn - that's why the metric is overlap, not count.
• Recall - keyword control (Q5) - % of the mechanically verified key found (the key for "which calls mention Competitor X" is a deterministic text scan; no judgment involved).
• Recall - build-in-house (Q1) - % of the answer key found. The key is the union of every system's candidates, each anchored to a verbatim transcript quote, adjudicated by a three-judge LLM panel, with splits ruled by a human. It closes only after all systems run - so this row stays "pending" until the final report, for every system including ours.
• Cost / question - metered marginal dollars to answer the question once, wherever the spend actually lands (warehouse AI credits, API tokens, product billing). Seat subscriptions are noted in the text but not counted as marginal cost.
• Time / question - wall-clock from prompt submitted to final answer, including any enrichment-job or queue wait. Async systems report submit→complete.

I expected the agent to beat chat on coverage. It did — sometimes. On the aggregate question it read nearly everything; on the rare-event question it read 1%, same as chat. Averaged out, it reads more (31% vs 10%) and repeats itself more (52% vs 24%) — real improvements — but it buys them with 3× the wall-clock, sub-agents that die mid-run, and a habit of reaching for pre-computed data the moment you stop watching.

The agent is a better sampler. It is not a system that reliably reads your data, answers the same way twice, or does it fast. The gap between "an agent that can loop" and "a system you'd trust with a churn number in a board deck" is still the whole thing — and, once again, the agent itself kept trying to close that gap by reaching for pre-computation.

Part 3 — the naive API

No agent, no chat, no warehouse LLM — just my own code stuffing as many transcripts as fit into one big call. The first arm where I control exactly how much gets read, and the first hard ceiling of the series.

Methodology appendix: Claude Code, Opus 4.8 high effort, memory off, one fresh session per run, all runs against the frozen BENCHMARK_CORPUS_1K. From Q3 onward the prompt was pinned to the corpus table only, after an unpinned run silently joined a pre-computed enrichment table (Finding 5); that run is logged and excluded. Coverage and architecture verified from Claude Code's own session logs (JSONL), not terminal capture. Full run log available.