Eval study notes · Coding practice set

Eval study notes

The statistics and engineering behind evaluating language models, situation first. Companion to the eval explainer videos, the same way the coding practice set goes with the coding videos.

Start hereWhich tool, when covers the three different things called “p”, where the probability in a confidence interval comes from, when to use McNemar, when to use Wilson, and one table that maps each situation to its tool.

Videos

Full video: eval statistics and platform in twelve chapters (46:05)

Watch on Google Drive (private to the owner) · Context file: the whole narration plus what is on screen, as one text file. Attach it to a chat with Claude or ChatGPT and ask about any part. Drive has no chapter menu, so drag to the time below; each chapter also shows its title on screen.

Pilot: four ideas (14:50), superseded by the full video

Watch on Google Drive (private to the owner) · Context file: the whole narration plus what is on screen, as one text file. Attach it to a chat with Claude or ChatGPT and ask about any part.

Statistics behind an eval number

  1. A score is an estimate: how precise is one number?
  2. Comparing two models: is the gap real?
  3. Power, sample size, pass@k: can this benchmark detect the change at all?
  4. Many comparisons: which of fifty movements are real?
  5. Judges, agreement, rankings: can you trust the grader?
  6. Benchmarks wear out: contamination, saturation, online tests

All the formulas on one page

Engineering an eval platform

  1. What evals are for
  2. Anatomy of an eval
  3. Graders
  4. Throughput and scheduling
  5. Degradation and on-call
  6. Designing an eval platform