Evals for agent analytics and testing

Measure Your Agent'sQuality Reliably

Checking every trace by hand doesn’t scale. TwoTail grades every one, with judges calibrated by your team and chosen for what moves your business, so you spot quality issues before your customers do.

Book a quality kickoff

The problem

Your agent makes mistakes. Your customers notice first.

Support agent

A customer asks the same question three times, then gives up.

Recruiting agent

It scores a strong candidate 4/10, and only a recruiter would spot it.

Lead-gen agent

It books meetings, but not the ones your customer wanted.

Research agent

You change the prompt, the model or a tool. One thing gets better, and two quietly get worse.

How it works

Your vibe-coded monitoring doesn't cut it.

TwoTail is the agent quality system your growing agent startup deserves. It's built on evals: automated checks that grade every trace.

Judged

Monitor quality in production

Hear about issues as they start, not when customers complain.

Every trace scored as it comes in. A Slack alert when a score moves.

Vibe-coded monitoring:

A prompt someone wrote once, never checked against your experts
Needs rewriting every time the agent changes
Flags non-issues and misses the failures nobody wrote down

With TwoTail:

Judges calibrated to your experts, not a prompt someone wrote once
Kept up to date as your agent changes
Always searching your traces for new use cases and failure modes

# agent-quality

TwoTailAPP9:41

repeat_question is up on support agent: 11% of traces today, from 4% last week. Mostly billing_dispute.

11% of traces
last 7 days

View 14 tracesMute

Calibrated

Align judges to your experts

Scores your expert agrees with, and a number for how often.

Your expert labels the few traces the judge is least sure about. The judge learns from them.

Vibe-coded monitoring:

Your expert scrolls hundreds of traces, or nobody looks at all
An observability tool built for engineers, not for labelling
No agreement rate, so you never find out whether the judge is right
Labelling takes a week, so it never happens

With TwoTail:

We pick the few traces worth your expert's time
A viewer built for labelling, not an observability tool built for engineers
Agreement tracked, so you know when a score can be trusted
Labelling takes an hour, not a week

candidate_score · 10 of 14 traces labelled

Expert✓✕✓✓✕✓✓✕✓✓

Judge✓✕✓✓✓✓✓✕✓✓

91%agreement with your expert
up from 64% before calibration

Proven

Measure what moves the business

Know which issues hurt your outcomes, and which to ignore.

Eval scores tested against retention, conversion or your customers' results, ranked by impact.

Vibe-coded monitoring:

Flags the visible problems, which are often the wrong ones
A frustrated power user gets flagged; a quiet user who churns goes unnoticed
You fix what is loudest, not what costs you

With TwoTail:

A map from quality to outcomes, tested statistically
Hypotheses ranked by impact, with what to do about each one
The first thing you fix is the thing that moves the business

shopping assistant · northstar: revenue

Tested

Test before you ship

Know whether a change helps before it ships.

Test sets built from production traces, graded by the same judges.

Vibe-coded monitoring:

Generic test sets built from capabilities, not from what customers do
Graded by checks nobody has calibrated
Drifts from production, so the tests pass and production still breaks

With TwoTail:

Test sets drawn from production and selected by impact
Graded by the same judges you trust in production
Updates as production changes, so a pass means something

model A vs model B

citation_accuracy48 / 48

tool_loops3 new fails

answer_length+31%

Meet the founder

Your Forward Deployed Analyst

I’ve worked for some of Europe’s highest growth scaleups, and founded 3 AI startups. When you use TwoTail, you get me too. A powerful tool combined with an experienced operator figuring out the perfect way to measure and optimize your agent.

Read the full story

What customers say

We used to rely on a Claude routine to find issues with our agent, but it was unreliable and I started ignoring it. Now we have evals to measure when agent quality drops - and we are confident that when it pings us on Slack it’s not a false positive.

Harold
Founder, Sliq

Tim is a deep thinker and brilliant problem solver, I would highly recommend him to anyone trying to solve a multi-faceted issue that may seem daunting, he has the craft to sort it out!

Xavier
Founder, DarkMatter

Timothy is an expert in data modeling. With his help, we now have a clear understanding of how users behave. We highly recommend working with him.

Randy
Founder, Recharge360

Because they have so much experience in the field, they proactively find the edge cases and implement systems so those are handled correctly.

Marlon
Founder, Assembly

Integration

Ten minutes, and no code changes.

TwoTail reads the traces you are already producing. Point OpenTelemetry at it, or turn on the Langfuse integration and leave everything where it is.

Direct, via OpenTelemetry

Standard OTLP/JSON from any framework, straight to the TwoTail API. As easy as pasting the setup prompt into your coding agent.

Through Langfuse

Already tracing with Langfuse? Turn the integration on and TwoTail reads the same feed. No code changes, nothing moves.

Works with your stack OpenTelemetry Langfuse LangChain OpenAI Agents SDK Claude Agent SDK Vercel AI SDK

Common questions.

You can, and you should. Claude Code reads traces, proposes failure taxonomies, and writes eval criteria well. What it cannot do is tell you whether its judgement is right: no ground truth from you, no agreement rate, a fresh taxonomy on every rerun, and a sample rather than your whole corpus. TwoTail is what it calls to get those. Your agent does the analysis; TwoTail makes it defensible.

Both, on the same judges. In production, TwoTail scores every session, not a sample, so you see what your agent is doing to customers today. Before a release, you run those judges over test sets built from real sessions. Most teams start with production, because that is where the surprises are.

Langfuse gives you somewhere to put scores. It has no opinion about which scores you should have, or whether they mean anything. TwoTail adds the two parts it does not have: labelling queues that pick the few sessions your expert needs to see, and a map from your scores to the business outcomes they move. Most of our customers run TwoTail on top of Langfuse: same OTel feed, different job.

Not out of the box. In specialised fields, frontier judges agree with you as little as 64% of the time. Calibrated against a few hundred well-chosen labels, they hold up, and the agreement rate is visible before you trust a single score.

Your data is stored in isolated Postgres databases with row-level security. Each account's data is fully segregated, and all connections are encrypted in transit.

Plans are priced for growth-stage teams, with a free trial once you're set up. It starts with a quality kickoff. Full details on the pricing page.

How we start

Start with a quality kickoff.

30 minutes on how to measure quality for your agent. You leave with a plan: what to measure, what good looks like, what to fix first. Yours to keep, whatever you decide.

If it's a fit, we set it up for you within two weeks.

Book a quality kickoff