An LLM-as-a-judge eval uses one LLM call to score the quality of another model’s output. It is a powerful technique for measuring AI quality, but a judge is itself an LLM system that can drift, overfit and carry bias. Here are 10 characteristics that separate a reliable judge eval from a misleading one.

What’s a Judge Eval?

The model as a judge pattern (better known as LLM-as-a-judge) is simply using one LLM call to judge the quality of another.

For example, let’s imagine we want to measure if an agent is answering a user politely. We’d have a sample of inputs and responses, e.g.

User: Where’s my order? It’s been two weeks and nobody is answering me.

Agent: I’ve checked, your order shipped on the 14th and is currently held at the local depot. I’m sorry nobody got back to you sooner; I’ve flagged it for release today and you’ll get a tracking update within the hour.

So we ask the “judge” to say whether they were polite or not.

The judge prompt might look something like this:

You are evaluating whether a customer-support agent responded politely.

Politeness means: the response is respectful in tone, does not blame or
talk down to the user, and acknowledges the user's situation before
moving on to the answer.

Read the conversation below. First, briefly explain your reasoning.
Then give a verdict: Polite: YES or Polite: NO.

User: {user_message}
Agent: {agent_response}

Resulting in a response like this:

Reasoning: The user is frustrated. The response acknowledges the delay,
apologises once without grovelling, and gives a concrete next step.
No blame is shifted to the user. Tone stays warm and direct.

Polite: YES

This is a medium complexity case. Politeness is subjective, but relatively easy to define. But even in this case, you’ve probably spotted that there are some important design decisions baked into how we write the judge prompt. So, let’s dig into the techniques for doing this in a reliable way!

What Makes a Judge Eval Good?

Let’s go through 10 characteristics of a good eval, from the simplest to the most complex.

1. Judge if it needs a Judge

Before you start, consider if your eval should actually be a judge. If this is something that could be calculated with a simple function e.g. “how long is the response?”, probably a “code eval” will be easier and cheaper!

2. One judge, one criterion

There’s an art/science to deciding what to measure and why. But given a certain topic, resist the temptation to measure everything at once!

So instead of something like this:

Score this input 1-10 based on X, Y, Z

… rather create 3 separate evals, one for each of X, Y, Z. It allows the LLM to concentrate, and makes charting and analysis more actionable. Nothing prevents you from aggregating them up later, it just doesn’t need to happen here (if you want to save cost, caching can help).

One judge, one criterion: three separate judges for X, Y and Z instead of one combined 1-10 score

3. Binary is Best

You need to choose a sensible scale for your eval, and more often than not binary is better than Likert (an ordered numbered scale).

LLMs are known to find numbered scales more challenging, and so do humans so it’ll make annotation harder (see below).

Having said this, I do find scales appropriate when there is a clear gradation of a single quality. “Completeness” runs naturally from missed-everything to covered-everything, so a 1-5 means something.

Just don’t use them when the levels are distinct qualities of the thing you’re measuring.

4. Concrete Rubric

For LLMs, responses come from context and instructions. For judge evals this means being explicit about your criteria: what counts as a pass, and what fails.

We call this a rubric, essentially a set of rules for how inputs should be scored.

For example, for our politeness eval:

Polite: YES only if all of these hold:
1. Acknowledges the user's situation before giving the answer
2. Places no blame on the user
3. No sarcasm or condescension ("as previously stated", "simply", "kindly")

An apology is not required. Brevity alone is not rude.

Writing a good rubric is probably an article in itself, but here’s the test: if two colleagues could read it and reach different verdicts on the same response, it’s not concrete enough.

5. Evolve from Examples

There’s a cheat code to coming up with good rubrics: start from real examples. Take a sample of your traces, score them in your head, and ask yourself which qualities separated the good from the bad.

For example, for politeness, let’s imagine we had these responses:

We can infer some criteria: the first acknowledges the situation before answering; the second jumps straight to an instruction. And “kindly” is an interesting false-positive - it reads as passive-aggressive, which tells you evaluating tone is as important as vocabulary.

You can of course include examples (“few-shot”) inside the judge, but I prefer if you can generalize them into specific rules - it forces the exercise and examples can lead to judges overfitting or hallucinating.

6. Reason before Verdict

This one’s really simple, but it turns out to be important. When telling your judge the response format, ask for the reason before the score. Research shows generating the reason first affects the score after it, while the inverse is more prone to inventing a reason to justify a guess.

Reason before verdict: asking for reasoning first produces a more reliable score than scoring first and justifying after

7. Watch out for Bias

There are a few known biases of judge evals, including:

It’s possible to mitigate all of these once you know them.

Additionally, it’s worth thinking about (and experimenting with) what strength of model is required for the evaluation - this will often be a cost / quality trade-off so which you should choose depends on the nature of the data and the volume.

8. Calibrate!

Even a well written judge will have systematic tendencies or blind spots.

For this reason we do calibration: comparing the scores of a model judge vs a human judge, and adjusting the model for agreement.

I’ll cover this process in more detail elsewhere, but fair to say human labelling is tiring, and that’s why I’m building statistically-determined prioritization of annotation queues inside TwoTail!

9. Recalibrate!

A mistake many teams make is to calibrate a judge just once. You’ll have something that works for your test cases, or perhaps a batch of production traces. But things change.

A new model version can start generating different outputs.

A new use case means the judge is seeing inputs that it has no guidance for in the rubric.

The solution is recalibrating often, working from an annotated sample of cases generated on a regular cadence. This ensures your judge is course corrected for change.

Recalibration: a judge calibrated once drifts as models and inputs change; recalibrate on a regular cadence from fresh annotated samples

10. Judge Systems

Over time, you’ll have a suite of evals. And for each of those an annotation and calibrating process. You’ll start to think of this more as a system, asking questions like:

These systems require active development and maintenance, but if they’re at the heart of your agent’s quality, it’s well worth doing properly.

Need Help with Evals?

I’m the founder of TwoTail, an eval analytics tool. Whether you’re writing your first evals, or you have evals and are trying to figure out if you’re doing it all right, I’d be happy to have a chat. Book a call at twotail.ai