Home/ Articles/ LLM as a judge in Langfuse: setup, cost, and how to know it's right

Guides

LLM as a judge in Langfuse: setup, cost, and how to know it's right

How Langfuse runs LLM-as-a-judge evaluators on live traces and experiments, what they cost, how to check them against human labels, and where that stops.

Timothy Daniell · Published November 9, 2026 · 7 min read

Key takeaways

Langfuse runs LLM-as-a-judge evaluators on live production data and on experiments. Each one returns a numeric, categorical or boolean score with the judge's reasoning, attached to the observation it scored.
Use observation-level evaluators. Trace-level evaluators are deprecated, and on Langfuse Cloud they stop producing results after 16 Nov 2026.
Judge calls go through your own LLM connection. Langfuse's docs estimate $0.01 to $0.10 per assessment, and every score counts as a billable Langfuse unit.
Score Analytics (beta) compares a judge score with a human score: Cohen's kappa, F1, agreement and a confusion matrix for categories, Pearson, Spearman, MAE and RMSE for numbers.
Agreement is a check on labels you already have. Langfuse doesn't choose which traces to label, or correct the judge's rate for its own errors.

Langfuse runs LLM-as-a-judge evaluators for you. You write or copy a judge prompt, map it to fields on your traces, and attach it to a rule. Langfuse then scores matching production observations, or experiment runs, and stores a score with the judge’s reasoning. Score Analytics will tell you how often that score matches a human’s. Whether the judge is right enough to trust is still on you.

How LLM as a judge works in Langfuse

A judge is an LLM call that grades another one. In Langfuse, the judge sees the input, the output, your rubric and, optionally, a reference answer, and returns a score with its reasoning. Scores can be numeric, categorical or boolean. The evaluation runs on Langfuse’s side, so nothing runs inside your application.

Four pieces make it work:

Evaluators. The judge itself: a prompt with variables like {{input}}, {{output}} and {{ground_truth}}, a model, and a score type. You start from a blank prompt or a Langfuse template. A template gives you an editable copy and leaves the original alone.
Where they run. On live production data, through rules that use observation filters to pick what to score, with a sampling rate to control cost. And offline, on experiments over datasets, before you ship a change.
Scores. Each result attaches to the observation it scored, one score per observation per evaluator, with the judge’s reasoning. Every judge run is also traced, in its own langfuse-llm-as-a-judge environment, so you can inspect it like any other trace.
Annotation queues. Where people label the same data by hand. Hobby gets one queue, Core three, Pro unlimited.

One date matters. Langfuse now recommends observation-level evaluators. Trace-level evaluators are deprecated, and on Langfuse Cloud they stop producing results after 16 Nov 2026. If your judges were set up at trace level, move them.

Setting one up

The documented path is short:

Add an LLM connection. The judge calls run through a provider connection you configure, so they use your model and your key.
Create an evaluator. Choose a template or a blank LLM-as-a-Judge prompt, pick the model, write the rubric and choose the score type.
Map the variables to fields on the observation, for example the root observation’s input and output.
Test it on sample observations. Read the score and the reasoning before you save. This is the cheapest moment to catch a vague rubric.
Attach it to a rule, with filters and a sampling rate, or use it in an experiment.

On cost: Langfuse’s docs estimate $0.01 to $0.10 per assessment in model calls. Separately, every score is a billable Langfuse unit, the same as a trace or a span. A judge on every observation of a busy agent adds up on both bills, which is why sampling exists.

For what goes into the rubric itself (binary verdicts, one criterion per judge, reasoning before the verdict), see LLM as a judge.

Checking the judge is right

A judge you haven’t checked is a guess with a number on it. Langfuse gives you two ways to check.

Score Analytics. Under Scores > Analytics, in beta, you pick two scores of the same type, say a human label and the judge’s verdict, and Langfuse compares them on the observations that have both. For boolean and categorical scores you get a confusion matrix, Cohen’s kappa, F1 and raw agreement. For numeric scores you get Pearson and Spearman correlation, MAE and RMSE. If agreement is low, the docs suggest reworking the judge prompt against labelled examples and comparing again.

The calibration skill. Langfuse also publishes a judge calibration workflow that runs through a coding agent. You build a dataset of inputs with the labels you expect, the agent runs your judge prompt as an experiment, and it reports accuracy, or in advanced mode a confusion matrix with precision, recall, TPR, TNR and F1.

Both are useful. Both assume you already have the labels.

Where it stops

Agreement is a check, not a method

Neither tool decides which traces your expert should label or how many. Score Analytics compares whatever overlap you happen to have, and the calibration skill runs on a dataset you built yourself. Annotating every trace isn’t an option, so the choice of what to label is most of the work.

And agreement isn’t the number you report. If your judge flags 12% of conversations and you know how often it’s wrong in each direction, the real failure rate is something else, with error bars. Langfuse doesn’t calculate that corrected rate. Nor does it keep the check up to date as your judge, your model and your traffic drift. Without that, you have a pile of labels and a kappa from last month, not a measurement.

The outcome isn’t in Langfuse

One customer runs a sales agent whose success is measured in sends. Sends live in their product database. None of them are in the traces. Evals without outcomes are a fine starting point, and they’ll give you a well-behaved agent. To move the number the business cares about, you need to know which judges predict it, and that join sits outside Langfuse.

Reading the trace is part of judging it

Three founders have told me they couldn’t see what their agent was doing in Langfuse. They had the traces. They couldn’t read them. That matters for labelling: an expert judging a support conversation should read it as a conversation, with the moment it went wrong in front of them. Langfuse lets you build views like that, but configuring them is tiresome.

Where TwoTail fits

TwoTail is the measurement layer on top of Langfuse. It connects to Langfuse Cloud with a pasted API key and no code change, and reads the same traces. It picks the few traces worth your expert’s time, tracks how often your judges agree with them as the agent changes, corrects the score for the judge’s own errors, and tests scores against the outcomes your business runs on. If you self-host Langfuse, send the same spans to TwoTail over OpenTelemetry instead.

Comparing tools first? See LangSmith vs Langfuse, Braintrust vs Langfuse and Arize Phoenix vs Langfuse.

Sources: Langfuse LLM-as-a-Judge, Langfuse evaluation overview, Score Analytics, Annotation queues, Judge calibration skill, Langfuse pricing. Checked 9 Nov 2026.

Disclosure: I build TwoTail, so weigh the last section accordingly.

Summary

Set up an LLM connection, create an evaluator from a template or a blank prompt, map its variables, test it, then attach it to a rule.
Sample live traffic to keep judge costs down. Each run is itself traced, in the langfuse-llm-as-a-judge environment.
Label a set of the same observations by hand in an annotation queue, then compare the two scores in Score Analytics.
Langfuse gives you the metrics. The method, which traces to label, how many, and what the corrected number is, stays with you.
The outcome your agent exists for usually isn't in Langfuse, so a well-scored agent and a valuable one can still be different things.

Frequently asked questions

Does Langfuse support LLM as a judge?

Yes, on every plan, including the free Hobby plan. You create an evaluator from a Langfuse template or your own prompt, pick the judge model, and attach it to a rule that selects which observations to score. It also runs on experiments.

How much does LLM as a judge cost in Langfuse?

Two parts. The judge calls go through the LLM connection you set up, and Langfuse's docs estimate $0.01 to $0.10 per assessment. Then each score is a billable unit on your Langfuse plan: Core is $29 a month for 100k units, with overage from $8 per 100k. Sampling keeps both down.

Can I check my Langfuse judge against human labels?

Yes. Score the same observations by hand, for example through an annotation queue, then open Scores > Analytics and compare the human score with the judge score. Score Analytics (in beta) shows a confusion matrix, Cohen's kappa, F1 and raw agreement for boolean and categorical scores.

Can I use my own judge prompt?

Yes. Start from a blank LLM-as-a-Judge prompt, or copy a Langfuse template and edit it. The prompt uses variables like {{input}}, {{output}} and {{ground_truth}}, which you map to fields on the observation, and you choose a numeric, categorical or boolean score.

Does TwoTail replace Langfuse evals?

No. TwoTail sits on top. It reads the same traces from Langfuse Cloud with a pasted API key, and adds the measurement Langfuse leaves to you: which traces your expert should label, agreement tracked as the agent changes, and a score corrected for the judge's errors.

Proven evals for your AI agents.

Your experts define what good looks like. TwoTail builds evals calibrated to their judgment, then shows which ones predict your business metrics.

Book a demo

Timothy Daniell

Founder of TwoTail. Building eval analytics for teams shipping AI agents to production.