Each of these starts with raw traces and ends with a shipped change. Here's how the loop plays out in practice.
Case 01 · Marketing platform · ad-copy generation agent
A marketing platform ran a dozen evals against its ad-copy agent but had no idea which ones tracked real quality. TwoTail found the two that predicted whether a human editor approved the copy, and showed the rest were noise.
The team rebuilt their evals as scoring rubrics: the quality dimensions they cared about (brand voice, factual accuracy, length), each written down, plus the real business outcome, editor-approved.
An annotation queue picked the traces worth the team's time. As they labelled, the failures got names and a taxonomy: off-brand tone, unsupported claims, and bloated copy, sized by how often each showed up.
With judges calibrated on those labels, every trace carried scores worth trusting. One view put each eval against editor approval: which ones move with the outcome, and which just add noise to the scorecard.
Two evals predicted the outcome; the rest came off the scorecard. The survivors became the live suite: always running, and worth trusting because they tie to the metric the business already cares about.
From 12 evals down to the 2 that predict editor approval. The live suite now scores what actually correlates with quality, not metrics that were quietly measuring nothing.
Case 02 · Customer-support chat agent
A support agent's overall success rate looked healthy. Segmenting by what users actually asked for revealed one intent dragging the average down, and a retrieval gap underneath it.
Segmentation grouped raw conversations into the intents users actually arrived with: FAQ, how-to, billing, refund, and ambiguous, no manual tagging required.
A calibrated resolution judge graded every conversation, so the rates were worth reading. The aggregate looked healthy. Broken out by intent, one segment stood far below the rest.
Pulling the refund failures into the taxonomy and comparing them to the wins exposed the structural difference: failing runs retrieved irrelevant documents, the winners didn't.
The diagnosis came with a direction: add query rewriting for refund-shaped questions and a dedicated refund-lookup skill. And because refund resolution is now an eval, the same suite that found the problem measures the change.
The one intent dragging the average down, and the retrieval gap causing it, found in an afternoon, not a quarter of dashboard-staring.
Case 03 · Research-assistant startup · multi-step web-research agent
A research-assistant startup's agent calls an LLM at every step to choose its next source. They wanted to cut spend on that per-step call without dropping task completion. Calibrated evals meant they could change the model and trust what the needle said.
Segmenting spend by step and model pointed straight at the culprit: the per-step source-selection LLM call, run once for every hop, dominated the bill.
Task completion was already an eval. Annotating a sample calibrated the judge against their own reading of the traces, and turned the raw score into a corrected baseline with an interval.
With the measure trusted, the source-selection call moved to a lite model. The calibrated judge scored both sides of the change, putting cost next to completion.
| variant | completion | cost / run |
|---|---|---|
| baseline · flagship | 62% | $0.024 |
| lite model | 59% | $0.014 |
Segmenting the results confirmed the saving held on routine queries as well as the hard outliers, so it was real rather than a sampling artifact.
−42% cost on the source-selection step, completion within 3 points of baseline, shipped with the receipts to back the call.
Setup in 10 minutes. First insights within a week.