LLM as a Judge

LLM as a Judge

Step 1 of 9

Two answers, one rubric, and no string match that could settle it. So a model grades.

The Idea

When correctness cannot be checked by a string match, one model can grade another's answers against a rubric. It is fast and it scales. It is also a model, with its own failure modes — and those failures are correlated with the thing being graded.

Real-World Example

A competition judged by a former competitor. Quick, informed, and reliably kinder to the style they were trained in than to the one they were not.

The Tradeoff

Judges favour whichever answer came first, favour length and fluency over correctness, and rate work in their own house style highly. Mitigations are cheap: swap the order and average, hide which system produced what, demand a short reason before the verdict, and recheck agreement against human labels every release.

Your turn

Put the steps in the right order.

1. 1Grade the same pair again with the two answers swapped↑↓
2. 2Write a rubric that names what counts as a win↑↓
3. 3Spot-check the judge's verdicts against human labels↑↓
4. 4Show the judge both answers with their sources hidden↑↓

Check order

Mini quiz

1 / 3

Position bias in a pairwise judge means it prefers:

• AThe shorter answer
• BThe answer it wrote itself
• CThe answer with more numbers
• DWhichever candidate was shown first