Each of these starts with raw traces and ends with a shipped change. Here's how the loop plays out in practice.
Case 01 · Marketing platform · ad-copy generation agent
A marketing platform ran a dozen evals against its ad-copy agent but had no idea which ones tracked real quality. TwoTail found the two that predicted whether a human editor approved the copy, and showed the rest were noise.
The team rebuilt their evals as scoring rubrics: the quality dimensions they cared about (brand voice, factual accuracy, length), each written down, plus the real business outcome, editor-approved.
An annotation queue picked the traces worth the team's time. As they labelled, the failures got names and a taxonomy: off-brand tone, unsupported claims, and bloated copy, sized by how often each showed up.
With judges calibrated on those labels, every trace carried scores worth trusting. One view put each eval against editor approval: which ones move with the outcome, and which just add noise to the scorecard.
Two evals predicted the outcome; the rest came off the scorecard. The survivors became the live suite: always running, and worth trusting because they tie to the metric the business already cares about.
From 12 evals down to the 2 that predict editor approval. The live suite now scores what actually correlates with quality, not metrics that were quietly measuring nothing.
Case 02 · Customer-support chat agent
A support agent's overall success rate looked healthy. Segmenting by what users actually asked for revealed one intent dragging the average down, and a retrieval gap underneath it.
Segmentation grouped raw conversations into the intents users actually arrived with: FAQ, how-to, billing, refund, and ambiguous, no manual tagging required.
A calibrated resolution judge graded every conversation, so the rates were worth reading. The aggregate looked healthy. Broken out by intent, one segment stood far below the rest.
Pulling the refund failures into the taxonomy and comparing them to the wins exposed the structural difference: failing runs retrieved irrelevant documents, the winners didn't.
The diagnosis came with a direction: add query rewriting for refund-shaped questions and a dedicated refund-lookup skill. And because refund resolution is now an eval, the same suite that found the problem measures the change.
The one intent dragging the average down, and the retrieval gap causing it, found in an afternoon, not a quarter of dashboard-staring.
Case 03 · Research-assistant startup · multi-step web-research agent
A research-assistant startup's agent calls an LLM at every step to choose its next source. They wanted to cut spend on that per-step call without dropping task completion. Calibrated evals meant they could change the model and trust what the needle said.
Segmenting spend by step and model pointed straight at the culprit: the per-step source-selection LLM call, run once for every hop, dominated the bill.
Task completion was already an eval. Annotating a sample calibrated the judge against their own reading of the traces, and turned the raw score into a corrected baseline with an interval.
With the measure trusted, the source-selection call moved to a lite model. The calibrated judge scored both sides of the change, putting cost next to completion.
| variant | completion | cost / run |
|---|---|---|
| baseline · flagship | 62% | $0.024 |
| lite model | 59% | $0.014 |
Segmenting the results confirmed the saving held on routine queries as well as the hard outliers, so it was real rather than a sampling artifact.
−42% cost on the source-selection step, completion within 3 points of baseline, shipped with the receipts to back the call.
Case 04 · Sliq · outbound-campaign agent
Sliq's agent runs outbound campaigns for its customers: 50 conversations and hundreds of task traces a day. Harold wanted to know the moment a customer got stuck, so he could step in or fix the agent. His answer was a daily Claude routine over the Langfuse traces. It was verbose, half the flags were false positives, and it was probably missing real cases. Checking it ate a third of his week, back and forth with Claude, until it was almost as quick to read every trace himself. He kept iterating the prompt. The prompt was never the problem: there was nothing to check it against.
Frustration went from a one-line judge prompt to a rubric and a failure taxonomy, built from real Sliq conversations rather than a definition dreamed up in the abstract.
An annotation queue surfaced only the conversations the judge wasn't sure about. A handful of labels calibrated it, and a handful a week keeps it that way. Nobody reads 50 conversations a day.
The calibrated judge grades every conversation as it lands. The false positives that made the old routine unreadable are gone, so a flag means a real customer is stuck.
Every frustrated case posts a short Slack alert. Harold opens each one, talks to the customer or ships a change. The failure taxonomy tells him which: missing tools and capabilities, configuration trouble on new campaigns, data conflicts that need cleaning up.
A third of a founder's week reading summaries, half of them wrong, down to a few pings a day he acts on every time. First fix shipped: the comment-monitoring UX his customers were asking for without saying so.
Setup in 10 minutes. First insights within a week.