Offline eval
An eval run on a fixed dataset before release, in CI or on demand.
ExampleRun 120 saved prospect briefs through a new drafting prompt and compare pass rates before it ships.
Evals glossary
Every agent eval sits on three axes.
Online, offline, judge, grader, test, monitor. Teams use them interchangeably. Place an eval on where it runs, who grades it and what it looks at, and the names sort themselves out.
The three axes
Name an eval by all three.
“Code eval” is a grader type. “Online” is a location. “End-to-end” is a scope. Most confusion comes from treating three independent choices as one list. Any eval takes one value on each axis, so name it in full: an online, model-graded, trajectory-level eval.
Personalization judge = online · model · turn
Where it runs
- Offline
- Online
- Inline (guardrail)
Who grades
- Code
- Model (LLM-as-judge)
- Human
Scope
- Step
- Turn
- Trajectory
Where it runs
- Offline
- On a fixed dataset, before release, in CI or on demand.
- ExampleRun 120 saved prospect briefs through a new prompt before shipping it.
- Online
- On production traffic, after the fact, usually on a sample and asynchronously.
- ExampleA judge scores 5% of yesterday's sent messages for personalization.
- Inline (guardrail)
- Synchronous, in the request path. It can block, retry or fall back.
- ExampleBlock any draft that contains an unfilled
{first_name}token.
Who grades
- Code
- Deterministic logic: regex, schema checks, executing something, counting.
- ExampleMessage under 300 characters. Prospect email domain resolves.
- Model (LLM-as-judge)
- An LLM applies a written rubric to the output or transcript.
- Example"Does the opener reference something specific to this prospect?"
- Human
- A person labels the output. The gold standard, and the source of calibration.
- ExampleYou label 50 drafts pass or fail for "sounds like a person wrote it".
Scope
- Step
- One LLM call or tool call inside a run.
- ExampleThe research step called the enrichment tool with the right company domain.
- Turn
- One input and one output.
- ExampleGiven this brief, is the first draft on-brief?
- Trajectory
- A whole run or conversation, often with a simulated user.
- ExampleA simulated user asks for a campaign with no audience. Does the agent ask for one before sending?
Where your evals sit
One outbound agent, 12 evals.
Same agent, 12 evals, no two in the same cell for the same reason. The agent researches prospects, drafts messages, and talks to the user who set up the campaign.
Offline
Step
CODE
Enrichment tool called with the prospect's company domain
Replays saved research runs and asserts the tool argument matches the domain on file.
Turn
JUDGE
Draft stays on-brief for a saved prospect brief
A judge reads each draft against its brief before a prompt change ships.
CODE
Draft under 300 characters
A length count on every draft in the golden set.
Trajectory
JUDGE
Simulated user with no audience: agent asks for one before sending
A user simulator plays the campaign owner; a judge reads the transcript.
Online
Step
CODE
Research step returned at least three prospect facts
Counts facts in the research span of sampled production traces.
Turn
JUDGE
Opener references something specific to the prospect
Scores a daily sample of sent messages for personalization.
HUMAN
Weekly labels on 50 drafts to calibrate the opener judge
Your labels are the ground truth the opener judge is measured against.
Trajectory
JUDGE
Campaign-setup conversation ends with the user approving the plan
A judge reads whole setup sessions and checks the final turn is an approval.
HUMAN
Weekly read-through of 20 full sessions
Where failures nobody has written an eval for yet get found.
Inline
Step
CODE
Prospect email domain resolves before send
A DNS lookup in the send path. No resolve, no send.
Turn
CODE
Block any draft with an unfilled template token
A regex on the draft before it leaves the agent.
Trajectory
CODE
Hard cap on messages per prospect per campaign
A counter checked before every send in the campaign.
Reference vs reference-free
Online, there is no right answer to compare to.
A grader either checks the output against a known-good answer, or against a rubric. Which one you can use depends mostly on where the eval runs: someone has to have written the answer down first.
Reference-based
“Does the domain the research step extracted match the hand-checked CRM record?”
output: acme-logistics.com
reference: acme-logistics.com
PASS
- Compares to
- A known-good answer, written in advance.
- Usual grader
- Code: exact match, a set overlap, a numeric tolerance.
- Where it runs
- Offline, on a golden set where every input has its answer.
- The cost
- Someone writes every reference, and it only covers inputs you thought of.
Reference-free
“Does the opener reference something specific to this prospect?”
output: “Saw Acme just opened a Rotterdam hub…”
rubric: names a fact true of this prospect only
PASS
- Compares to
- A rubric or rule. There is no ideal opener to match.
- Usual grader
- A judge, or code for hard rules such as “no unfilled template token”.
- Where it runs
- Online, on live traffic. Also offline, for outputs too open-ended to have one answer.
- The cost
- A judge has to be calibrated against human labels before its pass rate means anything.
Judgement vs description
An intent tag is not an eval.
Some LLM calls over traces judge quality: pass or fail, a score. Others only label what happened: an intent, a segment. The second kind is a classifier. It is built the same way and does a different job. Classifiers slice the data; evals say whether it was good.
Our framing Neither source draws the line this way.
Evaluator
“Did the setup conversation end with the user approving the plan?”
PASSFAIL
Says whether it was good.
One trace
- campaign_setup
- user: “send to my list today”
- research
- draft_messages
Classifier
“What was the user trying to do, and from what starting point?”
intent: send_campaignsegment: no_audience
Says what happened.
Setup ends with approval: pass rate by segment
Capability vs regression.
Within judgement, Anthropic draws one more split. A capability eval asks whether the agent can do something yet, and should start with a low pass rate. A regression eval asks whether it still does what it did, and should sit near 100%. Capability tasks graduate into the regression suite once they pass reliably.
Glossary
28 terms, one definition each.
Grouped by axis, with the synonyms you will meet and a check on the same outbound agent.
Where
Online eval
An eval run on production traffic after the fact, usually asynchronously on a sample.
ExampleA judge scores 5% of yesterday's sent messages for personalization.
Guardrail
A check in the request path that blocks, retries or falls back before output reaches the user.
ExampleBlock any draft that still contains an unfilled {first_name} token.
Production monitoring
Tracking eval and product metrics on live traffic over time, with alerts.
ExampleAn alert fires when the weekly personalization pass rate drops below 80%.
Who grades
Code-based grader
A deterministic check written in code: regex, schema validation, execution, counts.
ExampleThe message is under 300 characters and the prospect's email domain resolves.
LLM-as-judge
An LLM that grades an output or transcript against a written rubric.
Example"Does the opener reference something specific to this prospect?", scored pass or fail.
Human evaluation
People labelling outputs, used as ground truth and to calibrate judges.
ExampleYou label 50 drafts pass or fail for "sounds like a person wrote it".
Pairwise comparison
Grading by choosing the better of two outputs on a stated criterion.
ExampleA judge sees two drafts for the same prospect and picks the one with the more specific opener.
Reference-based
Grading against a known-good answer.
ExampleThe company domain the research step extracted matches the one in a hand-checked CRM record.
Reference-free
Grading against a rubric or rule with no known-good answer; the usual option online, where no ground truth exists.
ExampleA judge decides whether a sent message is on-brief. There is no ideal message to compare it to.
Scope
Single-turn eval
An eval of one input and one output.
ExampleGiven this prospect brief, is the first draft on-brief?
Step-level eval
An eval of one step inside a run, such as one tool call and its arguments.
ExampleThe research step called the enrichment tool with the right company domain.
End-to-end eval
An eval of a whole run or conversation, judged on the transcript, the outcome, or both.
ExampleThe campaign-setup conversation ends with an approved plan and a scheduled send.
User simulator
An LLM playing the user so a multi-turn eval can run without a real person.
ExampleAn LLM plays a campaign owner who wants to send today but has no audience list.
Outcome
The final state of the world after a run, as distinct from what the agent said happened.
ExampleThe agent says it queued 40 messages. The outcome is what the send queue holds.
Purpose
Capability eval
An eval of something the agent can't reliably do yet; a low pass rate is expected.
ExampleWrite a follow-up that answers the objection in the prospect's reply. It passes 30% of the time today.
Regression eval
An eval of something the agent already does; the pass rate should sit near 100%.
ExampleEvery draft keeps its unsubscribe line. It passes on every build, and a miss blocks the release.
Classifier
An LLM or code check that labels a trace with an intent or segment, with no judgement of quality.
ExampleTag each setup conversation intent: send_campaign and segment: no_audience.
Data
Task
One test case: inputs plus success criteria.
ExampleOne prospect brief, plus the criteria a good first draft has to meet.
Golden dataset
A hand-curated set of inputs, often with reference outputs, run on every change.
Example120 prospect briefs with approved reference drafts, run on every prompt change.
Trace, span, session
A trace is one request end to end; spans are its steps; a session is related traces over time.
ExampleOne draft request is a trace. Its research and drafting calls are spans. A user's week of campaign edits is a session.
Transcript
The full record of one run: messages, tool calls, reasoning, intermediate results.
ExampleEvery message in a campaign-setup conversation, plus each enrichment call and what it returned.
Practice
Error analysis
Reading traces, coding what went wrong, and grouping it into failure modes before writing evals.
ExampleYou read 100 sent messages, note each problem, and find a third of them open with the same generic line.
Judge calibration
Measuring a judge against human labels, usually by true positive and true negative rate, and adjusting it until they agree.
ExampleRun the opener judge on your 50 labelled drafts and measure how often it agrees on the passes and on the fails.
Judge drift
A judge's agreement with humans degrading over time as models or definitions change.
ExampleAfter a model upgrade, the opener judge starts passing generic openers it used to fail.
Eval harness
The infrastructure that runs evals end to end: execution, recording, grading, aggregation.
ExampleThe script that loads saved briefs, runs the agent, records each transcript and applies every grader.
pass@k and pass^k
pass@k is the chance at least one of k attempts succeeds; pass^k is the chance all k do.
ExampleA draft that is on-brief 80% of the time has a pass@3 of 99% and a pass^3 of 51%.
Saturation
An eval scoring near 100%, so it catches regressions but no longer shows improvement.
ExampleThe character-limit check has passed every draft for three months. It still guards, but it shows no progress.
How the pieces connect
Evals are a loop, not a suite.
- Production tracesLIVE TRAFFIC
- Error analysisHUMAN
- Write evaluatorsCODE OR JUDGE
- GuardrailsINLINE
- Calibrate judgesVS HUMAN LABELS
- Golden set and CIOFFLINE
- ShipRELEASE
- Online evals and monitoringONLINE
- New failuresFOUND IN PRODUCTION
- ↺ Back to error analysis
Offline evals stop the failures you already know about. Online evals find the ones you don't, and error analysis turns them into the next evaluator.
Common confusions
Six questions teams argue about.
Is a guardrail an eval?
It runs the same kind of check with a different job. A guardrail sits in the request path and can block, so it has to be fast and rarely block a good response; that usually means code, not a judge.
Is LLM-as-judge online or offline?
Either. The judge is the grader; where it runs is a separate choice.
What's the difference between a test and an eval?
In common usage, a test asserts one case passes and an eval measures a rate across many. Teams blur this; the axes don't care which word you use.
Is an intent classifier an eval?
No. It labels what happened without saying whether it was good. You need it to slice eval results, not to replace them.
How many examples does an offline eval need?
Anthropic suggests 20 to 50 tasks drawn from real failures to start. Hamel Husain describes 100 or more as typical for a CI golden set, though many teams start smaller.
If I run online judges, do I still need offline evals?
Yes, for different failures. Offline evals stop known failures before release; online evals find the ones you didn't know about. Hamel Husain is explicit that a CI pass rate does not predict production accuracy.
Sources
Where this comes from.
Demystifying evals for AI agents
Anthropic · Jan 2026
Hamel Husain
hamel.dev
Our framing The three axes as a naming device, and the split between judgement and description. Neither source draws them this way.
Anthropic uses “eval” for automated evals in development and treats production monitoring as a separate layer. Hamel Husain counts background monitoring and guardrails as ways an eval is put to work. This page uses the broader sense.
Online evals
Know how far to trust your judges.
TwoTail runs online evals on your production traces, with LLM judges calibrated against your own labels.