We read the notes from 5,000 panel reviews this spring and sat with four senior reviewers for an afternoon each. The quotes below are not transcripts. They are composites: lines we heard repeatedly, paraphrased into the voice of a single fictional reviewer who stands in for the group. The patterns are real. The wording is constructed. We chose this format because the quotes carry the texture of the work better than a paraphrase, and the texture is the part most engineers do not get to see.
Why this matters
Most engineers prepare for the rubric. The rubric is the scoring grid; it is not the read flow. The read flow is the sequence of attention a reviewer pays during the twelve minutes they have for your session. Two engineers with the same diff and the same audio defence can score a full point apart because one wrote for the read flow and one wrote for the rubric. The read flow rewards engineers who surface the decision early, name the trade-off plainly, and stop annotating after the second pivotal moment. The patterns below explain why.
What do you actually read in the first 60 seconds?
The first sixty seconds form a hypothesis. The hypothesis is not a score; it is a prediction the reviewer spends the next eleven minutes either confirming or breaking. A strong-session hypothesis sends the reviewer to the audio defence to confirm the engineer can articulate. A weak-session hypothesis sends the reviewer to the diff to find the bug they expect. Most reviewers do not realise they have a hypothesis until they have to revise one.
I open the replay, I read the one-liner, I count the decision points, and I already have a guess. The next eleven minutes are me arguing with my own guess. The best sessions are the ones that make me change it by minute three.
The composite captures something most engineers do not write for. Reviewers do not arrive neutral; they arrive with prior beliefs from the previous session, the coffee, the calibration sheet, and the visible shape of the replay timeline. The engineers who score best are the ones who land their first surprise early, so the reviewer's hypothesis is forced to update before the diff spot-check stage. Saving the cleverest decision for the end of the audio is a common, expensive mistake. By the time the reviewer hears it, the score is already forming.
If your first three annotation lines are about scaffolding, you have just told me to read your code like a draft. I will. And drafts score like drafts.
The fix is mundane. Write the first three lines of your annotation as if they were the abstract of the session. Name the constraint. Name the choice. Name the trade-off. Save the scaffolding for later — or do not annotate the scaffolding at all. Reviewers do not penalise quiet scaffolding. They penalise loud scaffolding that crowds out the decision.
When do you pause the replay?
A pause is a tell. Reviewers do not pause to admire; they pause to investigate. The pause means the reviewer is no longer reading and has switched to reasoning. That switch is expensive. Every pause longer than fifteen seconds is a line item on the reviewer's mental ledger, and the ledger feeds the defensibility score whether they intend it to or not.
I pause when the audio says one thing and the annotation says another. Not to penalise — to figure out which one you actually believed. Sometimes neither, and that is the worst answer.
The most common cause of a contradiction-pause is annotating before the audio defence rather than after. The engineer writes a confident annotation at the moment of the decision, then loses confidence on the audio and softens. The reviewer hears two engineers in one session. Neither one of them is the engineer being scored. Writing the annotation after the defence — or rewriting it after — closes the gap. About forty percent of contradiction-pauses we sampled traced back to annotation timing, not annotation content.
What's a signal that always blocks a pass?
A block is not a fail. It is a "this needs a second pair of eyes" call. Block-the-pass signals are the ones where the reviewer cannot, in good faith, score the session alone. The dominant block is the silent boundary: the engineer changed something that ought to be noticed and the replay shows no evidence they noticed. The reviewer does not know whether the engineer rejected the consideration or never had it. That uncertainty is what triggers the block.
If you cross the auth boundary without saying the word "auth" — in a comment, in a test name, in the audio — I am going to flag it. I am not deciding you were wrong. I am deciding I cannot tell whether you noticed.
The second block-the-pass signal is rarer but more consequential: a decision point that lists only what was accepted, never what was rejected. A decision is a choice between alternatives. A decision point that names no alternative is not a decision; it is a description. Reviewers cannot score defensibility on a description, so they block.
Tell me what you didn't pick. The thing you did pick is in the diff. The thing you didn't pick is the only place I can read your taste.
The structural fix is small. Every decision-point annotation should name at least one alternative — even one you rejected in two seconds — and the reason you rejected it. Two sentences. The second sentence does more work than the first.
What's a signal candidates overestimate?
Candidates prepare for the defence by practising confident speech. Reviewers, by the time they have done a hundred reviews, hear confident speech as a warning. The signal they want is calibrated uncertainty: "I chose A, but if the input distribution were B I would have chosen C, and here is the test that would have told me." That sentence scores higher than any version of "I chose A because A is right."
The candidate who says "I'm pretty sure but I'd check X" scores higher than the candidate who says "yes, definitely." Every time. I do not know why this surprises people.
The asymmetry is uncomfortable. Engineers are taught to interview by removing hedging. Panel reviews are not interviews in that older sense. They are reasoning audits, and an unhedged audit reads as either overconfident or rehearsed. The cheapest preparation move an engineer can make is to practise stating one calibrated uncertainty per decision and stopping there. No more, no less. Two unprompted uncertainties read as anxiety. Zero reads as bravado. One is the target.
How do you handle disagreement on a panel?
Panels disagree because the rubric does not score a single artifact. It scores six dimensions of reasoning evidence, and the evidence carries differently for different reviewers. Correctness is the dimension with the highest agreement because it is the dimension closest to the artifact. Defensibility is the lowest because it is the furthest. The split is not arbitrary — it tells you which dimensions reward writing for the read flow and which reward writing for the diff.
When two reviewers split by more than 1.5 points on a dimension, a third reviewer reads independently before either of the first two sees the third's score. The third reviewer breaks the tie about seventy percent of the time on their first pass. The other thirty percent goes to a structured panel discussion where the dimension rubric anchors are read aloud and applied line by line. We started the structured discussion in 2024 after noticing that defensibility splits resolved faster when reviewers anchored on the rubric language than when they argued about the engineer.
| Signal | What it tells the reviewer | What to do about it |
|---|---|---|
| Red: unexplained acceptance | The engineer cannot reconstruct the decision. | Name the rejected alternative and the discriminating criterion in two sentences. |
| Red: contradiction under defence | The engineer did not reconcile annotation with audio. | Annotate after the audio defence. Re-read the annotation against the audio before submit. |
| Red: silent boundary crossing | The engineer did not notice the auth or schema line. | Say the word 'auth' or 'schema' once at the crossing, in any medium. |
| Red: overconfident audio | The reasoning is rehearsed, not lived. | State one calibrated uncertainty per decision point. Stop there. |
| Green: position change under pushback | The engineer updates when evidence is good. | Practise updating with a peer for five rounds before a Hard-tier session. |
| Green: named alternative considered | The engineer chose, did not default. | Two sentences per decision point. One for what you picked, one for what you didn't. |
| Green: boundary called out in advance | The engineer noticed; reviewer does not have to. | Add a one-line comment or a test name at every auth or schema line. |
The full calibration study with inter-rater data, reviewer cluster analysis, and the dimension agreement table.
From Codritium Research
Panel Reviewer Calibration
What makes a panel reviewer reliable. Inter-rater dynamics, drift over weeks, the warm-up effect, and the strictness/agreement tradeoff that emerged once we ran 240 calibration sessions.
Common questions
Are these real quotes?
No. The quotes are composites — paraphrased lines we heard from multiple reviewers across 5,000 reviews and four sit-down sessions, attributed to a single fictional reviewer voice. The patterns are real; the wording is constructed to protect individual reviewers and to avoid implying any single reviewer carried any single view.
Do reviewers know each other?
Yes, by name and calibration record. They do not discuss live sessions with each other. They calibrate as a group quarterly on a fixed practice set, which is where most of the inter-rater agreement gain comes from. Reviewer rotation is quarterly to prevent stylistic drift.
How are reviewers paid?
Per review, at a rate calibrated against the median senior hourly for the reviewer's region. Reviewers are contracted, not employees. Pay does not vary with the score the reviewer gives, which is the standard protection against score-inflation incentives.
Can you appeal a reviewer's call?
Yes. An appeal triggers a third-reviewer pass with the original two reviewers' notes redacted. About 9% of appeals overturn the original outcome. Appeals are most often successful on defensibility splits and least often successful on correctness, which mirrors where reviewers actually disagree.
What happens if reviewers disagree by 1.5 points?
A third reviewer scores independently before they see either of the first two scores. If the third score breaks the tie, it stands. If the third score is also more than one point from both originals, the dimension goes to a structured panel discussion that reads the rubric anchors line by line. The structured discussion was introduced in 2024 and reduced unresolved splits by about a third.
What stops a reviewer from being biased?
Three things. The rubric anchors are concrete enough that reviewers can be coached when they drift. Inter-rater agreement is measured every quarter on a shared practice set and reviewers who fail calibration twice are rotated out. And appeals are read by an independent third reviewer with the originals redacted. None of these eliminate bias; together they make persistent bias visible quickly.
Where to go next
- The calibration study: Panel reviewer calibration
- The rubric reviewers score against: The defensibility rubric, 2026
- The earlier post this Q&A complements: What panel reviewers actually look for