Panel Reviewer Calibration

Why this matters

Every Hard-tier session on the platform passes through a human panel. Two reviewers score it. If they disagree, a third weighs in. Their scores end up in benchmarks, reports, certifications, and hiring decisions. Reviewer variance is the floor underneath every claim we make.

A year ago we treated calibration as an onboarding step — read the rubric, score five practice tasks, start reviewing for real. The data made that look thin. New reviewers disagreed with the established panel often, and the disagreement was not random; it tracked the reviewer's strictness, their domain background, and the rubric dimensions they weighted heavily. We rebuilt the program around what the data showed.

This piece reports what the data showed. Inter-rater dynamics across 47 reviewers, 240 calibration tasks, 11,280 paired scores. It is a meta-study about the people doing the panel review, not about the candidates being reviewed.

Methodology in one paragraph

A "calibration task" is a frozen Hard-tier session that the panel has already scored at least 12 times, used as a reference benchmark for new reviewers. A "paired score" is one of those scores set next to a baseline derived from the established panel's distribution. We measure agreement with Krippendorff's α (ordinal) computed weekly on the rolling 6-week window. "Strictness" is the reviewer's mean rubric score minus the cohort median, computed across the most recent 60 tasks they scored. The 47 reviewers in scope completed at least 30 calibration tasks between 2025-09-01 and 2026-03-31. Reviewers who completed fewer than 30 are excluded from the cluster and drift analyses but counted in the funnel.

The strictness-agreement tradeoff

The first surprise was that the strictest reviewers are not the most reliable. Neither are the most lenient. We plotted each reviewer's strictness (mean score minus cohort median, in rubric units of 0–5) against their inter-rater agreement (Cohen's κ against the established panel baseline). We expected a flat band. We got a clear inverted-U.

Reviewers near the cohort median sit at the peak — κ above 0.80. Reviewers a full rubric point above or below the median fall to κ near 0.45. That is not a small drop; at κ = 0.45 the reviewer is roughly as informative as a flipped coin biased in their direction.

The pattern has a mechanical explanation. A reviewer who consistently scores half a point higher than the panel still ranks sessions in the same order — they just shift the scale. But once strictness exceeds ~0.6 rubric points, the reviewer starts clipping the scale at the top or bottom: every Strong session looks 5/5, or every Weak session looks 1/5. Clipping destroys rank information. That is what the falling κ at the edges shows.

The practical move was to fold a calibration check into every 50 reviewed sessions. If a reviewer drifts more than 0.4 rubric points from cohort median in either direction across 30 consecutive tasks, they get a recalibration prompt. Drift is normal. Unmonitored drift is the problem.

Warm-up dynamics

A new reviewer's first scores look noisy because they are. They have not yet learned the panel's working interpretation of the rubric. They have read it, but reading is not enough. We tracked how α moves across the weeks following each new reviewer's first session.

The curve rises sharply through week 6, plateaus around week 8, and barely moves after week 10. The plateau is not 1.0 and we don't expect it to be. The cohort floor is around 0.78–0.80 because the rubric itself has irreducible ambiguity in a few dimensions, mostly Spec Fidelity and AI-Rejection, which require an interpretive judgment the rubric only partly constrains.

The implication for the program: shipping a reviewer onto live Hard-tier sessions before week 6 is too early. Their scores will count and they will be miscalibrated. We now hold new reviewers in a calibration loop until they hit α ≥ 0.75 on a rolling window of 20 tasks. The median time to that gate is 11.4 tasks; the long tail goes out to 24.

Reviewer clusters and what they emphasize

Not all reliable reviewers are interchangeable. Some weight Correctness heavily and treat Defensibility as a tiebreaker. Some do the opposite. We k-means-clustered reviewers on the relative emphasis they place on each rubric dimension (computed as their mean score per dimension, z-scored per reviewer, averaged across tasks). Four clusters fall out cleanly.

Correctness-first reviewers (n=11) score code that works and forgive everything else. Defensibility-first reviewers (n=14) read the replay first and the code second. Balanced reviewers (n=15) score every dimension within 0.1 of every other; they are the largest cluster and the most predictable. Anti-AI reviewers (n=7) score sessions higher when the candidate rejected an AI suggestion and lower when they accepted one without comment.

"Anti-AI" is a behavioural descriptor, not a value judgment. We name it as we see it. Their inter-rater agreement with the other clusters is the lowest of the four, but their internal agreement with each other is the highest. They are a coherent rating culture. The decision we landed on is to weight panels so no single cluster dominates a Hard-tier session — at least one of the two scorers must come from a different cluster than the other. The change moved cohort α from 0.74 to 0.79 over six weeks.

Where new reviewers fail first

Not all calibration tasks expose drift equally. A handful catch new reviewers more often than the rest. We pulled the 10 tasks where new reviewers' first-attempt scores deviated most from the panel baseline, weighted by how often each task is included in the warm-up rotation.

The top two — T-118 and T-072 — share a property. Both involve a "fix" that looks correct in isolation and is wrong against the broader system. The candidate's replay reveals the missing context. New reviewers who skip the replay and grade the code score these high. Calibrated reviewers, who watch the replay first, score them low. The 1.4-point gap is essentially a measure of how much replay-reading habit a reviewer has built.

The sparkline trends matter too. T-194 (stale schema migration) has drifted down over six quarters because candidates have gotten better at it — the AI-tooling AI ecosystem now flags schema staleness more aggressively, so we see fewer truly bad solutions. T-204 (AI-suggested injection) has drifted up because AI assistants now refuse the obvious injection patterns; candidates who get the bug shipped now are working around the AI's safety, which scores higher on AI-rejection. The rubric has not changed. The candidate population has.

Pipeline retention

The funnel from session-submitted to final-verdict has its own attrition shape. Not every Hard-tier session is panel-reviewable; not every reviewable one finds a second reviewer in time; not every two-reviewer pair agrees within rubric tolerance and avoids a tiebreaker.

Two attrition steps matter most. Submission-to-eligible drops 32% because ineligibility includes incomplete replays, candidates who opted out of panel review, and sessions where the platform's automated scoring flagged a confidence issue we'd rather resolve before a human spends time. Assigned-to-first-pass drops 5% because reviewers occasionally pass on tasks outside their domain confidence. The 7% drop from first-pass to second-pass is mostly latency — sessions that wait more than 14 days for a second reviewer are bumped out of the cohort and re-queued the following quarter.

Calibration history

The calibration program has a history shorter than the platform's but already informative. Each change moved cohort α by a measurable amount, and a few moved it the wrong way before we corrected.

The biggest single jump came from mandating replay viewing in Q2 2025. α moved from 0.66 to 0.71 inside one quarter. The smallest came from the warm-up gate — it had no impact on average α but cut the worst-end-of-distribution disagreements roughly in half. Different changes moved different parts of the distribution; reading the mean alone hides the work each change actually did.

Caveats

Reviewers volunteered for the calibration program. We did not assign them. There is almost certainly selection bias on conscientiousness and on existing rubric familiarity. The α numbers are the best we have but they reflect a self-selected cohort, not the population of all possible reviewers.

The 240 calibration tasks are a fixed corpus. They are well-loved; they are also possibly overfit. We refresh ~10% of the corpus each quarter, but a reviewer who scored well on the corpus in 2025 might be partly memorizing it. We have not yet run the experiment that swaps a reviewer's calibration corpus mid-quarter to see how much skill is corpus-specific.

α is one agreement statistic among several. We picked Krippendorff because it handles ordinal data and missing pairs cleanly. ICC or weighted κ would give numbers in the same range but with different sensitivities. The conclusions don't flip across choices but the absolute values shift by ~0.05.

The "Anti-AI" cluster name is provocative. We mean it as a behavioural descriptor — these reviewers reward AI-skeptical behaviour in candidates more than the other clusters do. We are not claiming they are right or wrong to do so. The empirical fact is that their scores cluster, and that balancing them across panels improved agreement without lowering signal.

What we'll do next

The next iteration recruits 20 additional reviewers from underrepresented domains (especially incident-response and read-path system design) to test whether the cluster shapes change. We will also rerun the warm-up curve with the new structured-replay annotations that landed in rubric v2.6 — early signs are that the warm-up plateau shifts up by ~0.03 because annotations remove some of the irreducible ambiguity in Spec Fidelity. Anyone running their own panel-review program is welcome to replicate the cluster analysis on their data; the k-means script is in our shared methods repo.