Research
Benchmarks we ran ourselves, with the data behind them: eight ways to analyze 1,000 conversations with AI, plus Snowflake load tests.
The Coverage Benchmark
Eight approaches to AI conversation analysis, each run on the same frozen set of 1,000 customer calls and five questions. One post per approach; the recap brings the findings together.
Chat + warehouse
Claude in the chat app, connected to the calls stored in a data warehouse, asked each question directly.
Autonomous agent
Claude as an agent with the same data, deciding for itself what to open, query and run.
One large API request
As many transcripts as fit into one model request, with the question asked over them in a single call.
Map-reduce
Code that sends every transcript through the model in batches, then combines the batch answers into one result.
Batch API
The same map-reduce code as Part 4, submitted as an asynchronous batch job at a lower price.
RAG
Retrieval-augmented generation: find the passages most related to each question and reason over only those.
Pre-computed enrichment
Claude reasoning over judgments already made about every call and stored as columns, instead of the transcripts.
Persisted classification (Rippit)
Rippit, run last: each question applied to every transcript as a new classification, with each verdict saved.
All 11 research reports
What It Takes to Trust an AI Answer About Your Customers
The series in one page: which approach read everything, which gave the same list twice, and what each means for a board number or a renewal list.
Can You Build Conversation Analysis in Snowflake Cortex?
We rebuilt it and load-tested to 200 users: no errors, but slow answers, full reads re-paid on every question, and eight hand-managed clusters.
What Happens When Claude Queries Raw Transcripts
Across chat, Cowork and Claude Code, Claude read about 0% of 1,854 calls and named 7, then 25, then 13 customers for the same question.
Same Count, Different Customers
Claude on a warehouse read 0.7–41% of the calls, built a different method each run, and quietly billed $15.80 of Cortex credits on one question.
The Agent Reads More, but Not the Same Way Twice
Claude Code read 95% on one question and 1% on another, averaged 11 minutes, and kept only half its account list on a re-ask.
“I Analyzed All 1,000.” It Read 99.
One large API call read about a tenth of the corpus, found 1 of 46 competitor mentions, and still claimed full coverage.
Reading Everything Isn’t Enough
Map-reduce read every call for about $61 a question, yet three runs named 73 accounts and agreed on only 16.
Half the Price, the Same Unstable List
Moving the same full read to the Batch API cut the cost by more than half; agreement across runs stayed at 29%.
Retrieval Finds Examples, Not Totals
RAG read a third of one percent of the corpus for $0.25 a question and found none of the accounts considering building in-house.
Judge Once, Reuse the Answer
Stored enrichment for all 1,000 calls fit one context window for $3.72 a question and produced a 67% overlap, limited by what the summaries preserved.
Every Call Read, Every Answer Saved
Question-specific classification read every transcript, found 45 of 46 control mentions and 52 build-in-house accounts, and returns the same saved list on a re-ask.
See what you can build
Every guide here ends in something you can run. Start from a ready-made agent, or describe your own and Rippit builds it.
