Calibrating an LLM judge: inter-rater reliability for model-graded evals

The problem with a judge score

This is the position most teams are in. They have an eval suite, it produces a score, the score moves, and nobody can say whether the movement is signal. Worse, the judge is usually a model that gets silently updated, which means the measuring instrument drifts independently of the thing being measured.

The fix is borrowed wholesale from a field that solved it decades ago. Content analysis, medical diagnosis and psychology all face the same problem — multiple fallible raters assigning labels — and the answer is an inter-rater reliability coefficient. The work is to apply it properly rather than to invent something.

Cohen’s kappa, and why not raw agreement

Raw percentage agreement is the intuitive measure and it is misleading, because two raters who both mostly say “pass” will agree constantly without either of them discriminating anything.

Two raters label 100 items. Both say "pass" 90% of the time, independently.

  observed agreement   p_o = 0.82        ← looks excellent
  expected by chance   p_e = (0.9 × 0.9) + (0.1 × 0.1) = 0.82

  κ = (0.82 − 0.82) / (1 − 0.82) = 0.00  ← they agree exactly as much as
                                            two coins weighted the same way

The 82% is entirely explained by both raters' preference for "pass".
It contains no evidence that either rater can tell pass from fail.

Computing it is a one-liner in any standard toolkit, which removes the last excuse for reporting raw agreement instead.

from sklearn.metrics import cohen_kappa_score

# Two label vectors over the SAME items, in the same order.
human  = ["pass", "fail", "pass", "pass", "fail", ...]
judge  = ["pass", "fail", "pass", "fail", "fail", ...]

kappa = cohen_kappa_score(human, judge)

# For an ordinal rubric (e.g. 1-5), near-misses should count partially:
kappa_w = cohen_kappa_score(human, judge, weights="quadratic")

Measure your humans first

This is the step almost everyone skips, and skipping it produces a specific and very common failure: a team concludes their judge is unreliable when in fact their task definition is unreliable, and they spend weeks tuning judge prompts against a target that was never achievable.

The canonical LLM-as-a-judge work makes exactly this point in the affirmative. Reporting on GPT-4 as a judge, Zheng et al. found it could match controlled and crowdsourced human preferences with over 80% agreement — which they note is the same level of agreement humans reach with each other. The claim is not that the judge is right; it is that the judge is as consistent as people are, which is the only standard available.

The fourth row is worth pausing on. A judge that agrees with humans more than humans agree with each other has usually learned a systematic quirk of whoever wrote the labels or the examples — which will look like excellent calibration right up to the point where the rubric is applied to something new.

The kappa paradox

This bites hard in practice because real production traffic is imbalanced. If 97% of your agent’s responses are fine, an eval set sampled uniformly from traffic will be 97% “pass” — and kappa on that set will be unstable and pessimistic no matter how good your judge is.

The practical consequence for an agent eval set: build it deliberately rather than by sampling. You want the failures over-represented, because the failures are the thing you are trying to detect — and a set that mirrors production will be dominated by the boring successes.

Which coefficient

For agent evals the honest default is weighted kappa, because most useful agent rubrics are ordinal rather than binary — a response is not merely correct or wrong, it is excellent, acceptable, poor or harmful. Unweighted kappa treats “acceptable versus poor” as exactly as bad a disagreement as “excellent versus harmful”, which is obviously wrong and quietly distorts everything downstream.

Known judge biases, and how to test for them

# Present each pair in both orders. A consistent judge gives the same winner.
flips = 0
for a, b in pairs:
    first  = judge(option_1=a, option_2=b)
    second = judge(option_1=b, option_2=a)
    # Normalise: 'option_1' in the second call refers to b.
    if winner_identity(first, a, b) != winner_identity(second, b, a):
        flips += 1

position_bias_rate = flips / len(pairs)
# Anything materially above zero means your pairwise scores carry an
# order artefact. Mitigate by always averaging both orders.

The last row of the table is the one that catches agent teams specifically. A judge asked to grade whether a multi-step trajectory was sensible is being asked to do the reasoning it is grading. If your hardest category is also where κ drops, you have found the limit of model-graded evaluation for that category, and the answer there is human review rather than a better prompt.

The calibration protocol

1. RUBRIC
   Write it as decidable criteria, not adjectives.
   Bad:  "the response is helpful"
   Good: "the response answers the question asked, cites the order ID,
          and does not promise anything outside the refund policy"

2. SAMPLE  (stratified, not uniform)
   ~300 items for a binary rubric; more for many categories.
   Deliberately over-represent failures and edge cases.

3. TWO HUMANS, INDEPENDENTLY
   No discussion until both are done. Discussion first produces
   agreement that measures conversation, not the rubric.

4. HUMAN-HUMAN κ  ← THE CEILING
   κ < 0.6?  Stop. Rewrite the rubric and re-label. Do not proceed
   to the judge; you have nothing to calibrate against.

5. JUDGE-HUMAN κ
   Run the judge on the same items. Compare against the ceiling,
   not against an abstract threshold.

6. RE-RUN ON CHANGE
   Any rubric edit, judge-model change, or judge-prompt change
   invalidates the calibration. Pin the judge model version.

Pin the judge model

A calibration is a statement about a specific judge model at a specific version with a specific prompt. If any of the three changes, the number no longer applies — and since providers ship model updates on their own schedule, an unpinned judge means your eval suite silently recalibrates itself without telling you. Pin the version, and treat a judge upgrade as a change that requires re-calibration before it ships.

Sample size and intervals

Report a confidence interval, not a point estimate. A κ of 0.72 from 40 items and a κ of 0.72 from 400 items are different claims, and only one of them supports a decision. Bootstrap the interval if a closed form is inconvenient — resampling the labelled set a few thousand times is cheaper than the meeting you will have about whether the number moved.

What to report

eval: refund_correctness         2026-07-30
  score .................. 78.4%  (n = 1,204)
  judge–human κ .......... 0.74   [0.68, 0.79]   ← weighted, quadratic
  human–human κ .......... 0.79   [0.73, 0.84]   ← the ceiling
  judge ................... claude-opus-5 @ prompt v4  (pinned)
  position-bias rate ...... 0.03
  eval set ................ stratified: 40% failure-weighted

Read as: the judge is performing near the achievable ceiling on this
rubric, so the 78.4% can be treated as a property of the system rather
than of the measuring instrument.

That final line is what calibration buys. Not a better score — the score is whatever it is — but the right to attribute it to your system. Without it, every eval discussion has an unresolvable ambiguity at the centre of it, and teams end up arguing about vibes with numbers attached.

Two connections worth making. Calibration needs labelled trajectories, which means it depends on having captured them — the evidence layer is a prerequisite, not a parallel workstream. And the reason trajectory evaluation is hard in the first place is the same reason agents are hard to test at all, which is the argument for building a workflow where you can.