# My LLM judge quoted the call. I couldn't find a third of its quotes.

> 35 flags from one scored call, checked against the transcript, and what the misses had in common

- Date: 2026-10-17
- HTML: https://www.sujit.top/notes/llm-judge-quotes

LLM judges are good at agreeing with people. In the MT-Bench paper, GPT-4 as a judge reached over 80% agreement with human preferences, the same level humans reach with each other ([Zheng et al., NeurIPS 2023](https://arxiv.org/abs/2306.05685)). That result is why so many of us now let a model grade other models, essays, support tickets and sales calls.

Agreement on a verdict is not the same as evidence for it, though. When I built [CallCoach-AI](https://callcoachai.sujit.top/), four LLM agents score an investor or sales call and every flag they raise comes with a quote: the exact words that triggered it. A founder clicks the flag and hears that second of audio. If the quote isn't in the call, the flag is worth nothing, however reasonable it sounds.

So I checked. This post is what one call taught me, with the numbers.

## The setup

The public sample report on the live app is a 13 minute 36 second call. The agents raised **35 flags** across four categories. For each flag, CallCoach searches the diarised transcript, one speaker segment at a time. The flag is marked *verified* only when the quote appears literally (ignoring case) inside a single segment. If it doesn't, CallCoach still places the flag at the segment that shares the most words with it, and labels it approximate.

**23 of the 35 quotes matched. 12 did not.** That is a 66% match rate, from a judge that writes fluent, specific, confident evidence every single time.

_[Figure: the 35 flags plotted by timestamp and category. Interactive version at the HTML link.]_

An unmatched quote is not automatically a lie. It can be a paraphrase, two sentences merged, or a transcription difference. But a quote the user cannot find is exactly the failure that breaks trust, so it is the one worth measuring.

## Where the misses were

My first guess was position. Long-context models are known to use information at the start and end of their input better than the middle ([Liu et al., "Lost in the Middle", TACL 2024](https://arxiv.org/abs/2307.03172)). A 13 minute transcript is long context, so I expected the middle of the call to be where quotes went wrong.

It wasn't:

| Part of the call | Flags | Matched |
|---|---|---|
| First third | 21 | 12 (57%) |
| Middle third | 7 | 7 (100%) |
| Last third | 7 | 4 (57%) |

One call is far too small to say the effect is absent, but it was clearly not what drove the misses here.

The category did:

| Category | Matched |
|---|---|
| Tension | 11 of 11 |
| Clarity | 8 of 12 |
| Compliance | 4 of 11 |
| Action item | 0 of 1 |

Compliance, the category with the highest stakes (refund rights, rate guarantees, cancellation windows), had the lowest match rate. Tension flags, which quote short reactions like a flat "No", matched every time.

## What the misses had in common

Two things showed up in the quote text itself, with no transcript needed to see them.

**They were longer.** Matched quotes averaged 11.8 words. Unmatched quotes averaged 19.8.

**They stitched turns together.** 4 of the 12 unmatched quotes join several speakers' turns into one "quote", the agent's question and the customer's answer run together as if one person said them. None of the 23 matched quotes do that. Because verification happens inside one speaker segment, a stitched quote cannot pass, even when every word in it was said. That is a design choice worth knowing about before you read the 66% as a hallucination rate: it measures evidence a human can find in one place, not truth.

That fits what compliance moments look like. They are long, formal sentences, often a question and a confirmation across two speakers. The model reconstructs the gist, which is what language models are trained to do, and the reconstruction no longer exists anywhere in the recording. Summarisation research found the same tendency years ago: human annotators found hallucinated content in all the model-generated summaries they reviewed ([Maynez et al., ACL 2020](https://arxiv.org/abs/2005.00661)).

## Two systems, two policies

I have shipped this check twice, with opposite policies, and the contrast is the useful part.

**CallCoach keeps the flag and says so.** An unmatched flag stays in the report, timed to its closest segment and marked as unverified. Nothing is hidden, and the reader decides how much to trust it. That suits coaching, where a flag with weak evidence can still start a useful conversation.

**FitNova drops it.** In [FitNova Call Intelligence](https://github.com/Blacksujit/FitNova-Call-Intelligence) the tagging prompt says every quote must appear character for character in the transcript, and the pipeline then enforces it: any tag whose quote is not an exact substring of the transcript is removed before storage and logged as a caught hallucination. The model is still free to judge; it is no longer free to invent the evidence. FitNova is a compliance tool, so a flag without proof is worse than no flag.

The same idea, applied to numbers instead of quotes, is how [Sentinel-AI](https://github.com/Blacksujit/Sentinel-AI) checks LLM output: it extracts each numeric claim, checks it against the source context, and corrects it when the source holds the right value.

## What this call suggests trying next

These come from the misses above. I have not measured them yet:

1. **Ask for one speaker's words per quote.** A finding that spans a question and an answer gets two quotes, each checked on its own. That would remove the stitched quotes by construction.
2. **Ask for the shortest span that proves the point.** Matched quotes were shorter; shorter evidence is also faster for a person to check.
3. **Track the match rate per category over time.** Compliance had the lowest rate here. A drop in that number is the earliest sign that a prompt or model change made the judge worse where it matters most.

## The trade-off

Dropping unmatched flags lowers recall. Some of them were probably real problems with a loose quote attached. Keeping them raises recall but asks the reader to do the checking. I don't think one policy is right; I think the policy should follow the cost of a wrong flag. For compliance, drop. For coaching, keep and label.

## Limits of this post

This is one call, 35 flags, one judge configuration. The category and length patterns are a reason to look harder, not a result. What I am confident about is the method: if your judge produces evidence, check the evidence mechanically, and measure how often it fails.

The sample report is public, so you can read every flag and quote yourself at [callcoachai.sujit.top](https://callcoachai.sujit.top/).

*I write about the seam between ML research and production systems.*