Eval study notes · Coding practice set
Exact-match graders are easy to trust because you can read their code. A growing share of what labs evaluate, from chat helpfulness to long-form reasoning to agent transcripts, has no single right answer, and is graded either by people or, far more often now, by another language model. An LLM judge is a measuring instrument, and the first question about any instrument is whether it measures what you think it does. This chapter covers how to answer that question with numbers, and then how labs turn pairwise judgements into a ranking.
A team has built an LLM judge that reads a model’s answer to a customer-support prompt and outputs PASS or FAIL. They hand you 200 answers that two human experts also graded, and the judge agrees with the human consensus on 80 percent of them. Is the judge good? What would you check before letting it gate a release?
Eighty percent sounds respectable, but agreement on its own mixes skill with luck. If the judge and the humans both say PASS most of the time, they will agree often even if the judge is guessing. Here are the actual counts: on 110 answers both said PASS, on 50 both said FAIL, on 25 the judge said PASS and the humans said FAIL, and on 15 the judge said FAIL and the humans said PASS. The judge says PASS 67.5 percent of the time and the humans 62.5 percent of the time. Two raters who said PASS at those rates completely at random would agree on 0.675 × 0.625 + 0.325 × 0.375 = 54.4 percent of items by chance.
Cohen’s kappa corrects for that. It asks how much of the possible improvement over chance agreement the judge actually achieved: κ = (observed agreement − chance agreement) / (1 − chance agreement). Here that is (0.80 − 0.544) / (1 − 0.544) = 0.56. On the commonly cited Landis and Koch scale, 0.41 to 0.60 is “moderate” agreement and 0.61 to 0.80 “substantial”. So this judge is moderate, not great, and the raw 80 percent oversold it.
Kappa has a well-known paradox you should be able to describe, because it bites eval teams constantly. When one label is rare, kappa can be low even when agreement is very high. Take a safety judge on a dataset where 94 percent of answers are fine: suppose judge and humans agree on 180 PASS and 4 FAIL, and disagree on 8 each way. Raw agreement is 92 percent, but kappa is only 0.29, because almost all of that agreement is on the easy majority class and the judge catches only a third of the failures the humans flagged. Kappa is telling you something real here: on the cases that matter, the judge is weak. The lesson is to report the confusion matrix and the per-class rates (how many of the human-flagged failures the judge catches, which is its recall on FAIL, and how many of its FAILs are real, its precision) next to kappa, never kappa alone.
For more than two raters, Fleiss’ kappa generalises the same idea, and Krippendorff’s alpha generalises further to ordinal or interval scales (a 1 to 10 rubric score) and to missing ratings. When the scores are numeric, a rank correlation such as Spearman’s ρ or Kendall’s τ between judge and human scores is also standard, and for system-level validation (does the judge rank models the same way humans do?) Kendall’s τ across models is the number people quote.
There is a step that teams skip and that interviewers notice. Before asking how well the judge agrees with humans, measure how well the humans agree with each other. If two expert annotators have a kappa of 0.6 between them, a judge with a kappa of 0.56 against their consensus is performing near the human ceiling, and further prompt engineering will not buy much. If the humans agree at 0.9, the judge has real room to improve. Inter-annotator agreement sets the ceiling for any automated grader and is also the best diagnostic of whether your rubric is well defined: low human agreement usually means an ambiguous rubric, which no judge can fix.
The literature since the 2023 “Judging LLM-as-a-Judge” paper (Zheng et al., the MT-Bench and Chatbot Arena paper) has catalogued a consistent set of biases, and a platform that offers judge-based graders should measure each of them as part of judge validation.
Position bias: in pairwise comparisons, judges favour whichever answer appears first (or, for some models, second). The standard mitigation is to run every comparison in both orders and either average or count only consistent verdicts; the rate at which a judge flips its verdict when the order is swapped is a direct measure of its unreliability. Verbosity bias: judges prefer longer answers even when the extra length adds nothing, which rewards models for padding. Mitigations include length-controlled win rates (AlpacaEval 2.0 introduced a regression-based length control in 2024) and rubrics that explicitly penalise unnecessary length. Self-preference: judges rate outputs from their own model family more highly, so using the model under test as its own judge is a conflict of interest. Sensitivity to formatting and style: markdown headers and bullet lists sway judges, which is why LMArena added “style control” to its leaderboard in 2024. And drift: if the judge is a hosted model, a silent provider update can shift every score, so a platform must pin judge model versions and keep a fixed calibration set that is re-scored whenever the judge changes.
That last point is a platform concern more than a statistics one, and it is the one most directly in the job description. A judge is a dependency like any other library. It needs a version, a regression test (the calibration set with human labels, re-scored on every change, with kappa tracked over time), and an alert when its agreement drops.
Many preference evaluations do not score answers in isolation; they ask which of two answers is better. Chatbot Arena (now LMArena) collects millions of such human votes, and internal evals often do the same with a judge. Turning a pile of pairwise wins into a ranking is the job of the Bradley-Terry model.
Bradley-Terry assigns each model a strength sᵢ and says the probability that model i beats model j is 1 / (1 + e^(−(sᵢ − sⱼ))), a logistic function of the strength difference. Fitting it is just logistic regression on the pairwise outcomes. The familiar Elo scale is the same thing with a change of units: a 400-point Elo difference means 10-to-1 odds. A useful conversion to keep in your head is that a 60 percent head-to-head win rate corresponds to about 70 Elo points, and 55 percent to about 35.
Elo as used in chess updates ratings sequentially after each game, which makes the result depend on the order in which games were played; that is fine for chess, where players change over time, but wrong for static models. LMArena moved from online Elo to fitting Bradley-Terry on all votes at once in late 2023 for exactly this reason, and computes confidence intervals on each model’s score by bootstrapping over votes. When two models’ intervals overlap heavily, their rank order is not meaningful, which is why their leaderboard shows rank ranges rather than single ranks.
The 2025 controversy around Meta’s Llama 4 on LMArena, where an experimental chat-tuned variant that was not the released model reached second place on the leaderboard, and the subsequent “Leaderboard Illusion” paper (Singh et al., April 2025) arguing that private testing of many variants lets labs select the best draw, (the paper counted 27 private Llama 4 variants tested before launch, and the released Maverick ranked 32nd where the tuned variant had ranked second) are both multiple-comparisons stories: test enough private variants against a noisy leaderboard and the maximum will look better than the model really is. If an interviewer brings up arena-style evals, connecting them back to Chapter 10 is a strong move.
Validate an LLM judge the way you would any measuring instrument: against human labels, with chance-corrected agreement (Cohen’s kappa for two raters, Fleiss or Krippendorff for more), and with the confusion matrix beside it because kappa collapses when one class is rare. Measure human-human agreement first; it is the ceiling. Test for position, verbosity, self-preference and style bias, run pairwise comparisons in both orders, and treat the judge as a pinned, versioned dependency with its own regression suite. Pairwise preferences become rankings through Bradley-Terry, which is Elo without the order dependence; bootstrap its confidence intervals, and remember that a leaderboard where labs can test many private variants rewards selection, not quality.