Eval study notes · Coding practice set
This page is for the night before. Each line is the tool, when to reach for it, and the number from the book’s running example so you can check your memory.
Standard error of a pass rate. √(p(1 − p)/n). At 72 percent on 500 questions: 2.0 points. Halving it takes four times the questions.
95 percent interval. p̂ ± 1.96 SE (Wald) is fine for a few hundred items away from 0 or 100 percent: 68.1 to 75.9 percent. Use Wilson by default (67.9 to 75.8 percent), and always near the extremes: 1 of 30 gives 0.6 to 16.7 percent where Wald gives a negative lower bound. Clopper-Pearson when you need guaranteed coverage.
Bootstrap. Resample the independent units with replacement 10,000 times, recompute the metric, take the 2.5th and 97.5th percentiles. Works for any metric. Resample clusters (passages, repos, environments), not items, when items share them.
Design effect for clustered items. Variance multiplies by 1 + (m − 1)ρ. Five questions per passage at ρ = 0.2: 1.8 times the variance, 1.34 times the error bar.
Comparing two models on the same items. Pair them. McNemar on the discordant counts b and c: exact binomial test of b against b + c at one half. Running example: 40 versus 55 gives p = 0.15, where the unpaired test on the same data gives 0.28. Paired SE of the difference: 1.95 points versus 2.8 unpaired.
Continuous per-item scores. Paired t-test on differences, or paired bootstrap, or permutation test (swap labels within items).
p-value. Probability of data at least this extreme if the null were true. Not the probability the null is true. Says nothing about effect size. Lead with the confidence interval on the difference.
Minimum detectable effect. About 2.8 standard errors of the difference (α = 0.05 two-sided, 80 percent power). Paired with discordance rate r: 2.8 √(r/n). At r = 0.2, detecting 2 points needs about 4,000 items; unpaired at 70 percent accuracy, about 8,000.
pass@k. Generate n > k samples, count c correct, per problem 1 − C(n − c, k)/C(n, k), averaged over problems. n = 20, c = 3: pass@1 0.15, pass@5 0.60. Never 1 − (1 − p)^k across a benchmark. Ask whether people mean pass@k, avg@k or maj@k.
Multiple comparisons. Forty null tests at 0.05: 87 percent chance of at least one false positive. Bonferroni α/m; Holm (sorted, α/m, α/(m − 1), …) is uniformly better; Benjamini-Hochberg (largest i with p₍ᵢ₎ ≤ iq/m) controls the false discovery rate and suits dashboards.
Cohen’s kappa. (observed − chance)/(1 − chance). 80 percent agreement with 54.4 percent chance gives 0.56, “moderate”. Collapses when one class is rare: 92 percent agreement, kappa 0.29. Report the confusion matrix and per-class recall beside it. Measure human-human agreement first; it is the ceiling.
Bradley-Terry / Elo. P(i beats j) = 1/(1 + e^−(sᵢ − sⱼ)). 400 Elo points = 10:1 odds; 60 percent win rate ≈ 70 points; 55 percent ≈ 35. Fit all votes at once, bootstrap intervals, show rank ranges.
Online tests. Randomise by user, analyse at user level, reduce variance with CUPED, do not peek without a sequential method.