A panel reviewer opens your replay at 9:14 on a Tuesday morning. They have a coffee, a calibration sheet, and twelve minutes budgeted for your session. The rubric tells them what to score. It does not tell them what to read first, what to skip, or how to weight the moment you changed your mind. Those decisions are not in the rubric. They live in the reviewer's read flow, and the read flow is what actually decides whether your session passes or gets flagged.
Why this matters
Most engineers preparing for a panel-graded session optimise for the rubric and ignore the read flow. The rubric tells you what gets scored; the read flow tells you what gets noticed. If your replay does not surface the decision points in the first sixty seconds, the reviewer will not find them later — they will find what looks like a clean diff and a quiet defence, and they will score from that. The difference between a 3.5 and a 4.5 is not in the code. It is in how the evidence is laid out for someone reading on a Tuesday morning.
What does a reviewer read in the first 60 seconds?
The first sixty seconds are the most important minute of the review. The reviewer is forming a hypothesis: this is a strong session, this is a weak session, this is a clean session with a hidden problem. The hypothesis shapes what they look at next. A strong-session hypothesis points the reviewer at the defence audio to confirm the engineer can articulate; a weak-session hypothesis points them at the diff to find the bug they expect. Both end up scored similarly in the end, but the path differs.
What gets read in those sixty seconds: the title and one-line summary of the session, the first decision point timestamp, the first three lines of the engineer's own annotations, and a glance at the final diff size. Nothing else. The reviewer is not yet reading code; they are reading the engineer's framing of what they did.
The timeline below shows the full read flow for a twelve-minute Hard-tier review. It is fixed across reviewers — we calibrated it during the v2.6 rubric refresh — and it is the flow we teach new panel members in their first week.
The single biggest piece of advice we give engineers preparing for a panel review is to write the first sixty seconds for the reviewer who has not had their coffee yet. Surface the two pivotal decisions. Name what the trade-off was. Do not bury them in the middle of a long annotation.
Which signals get someone flagged in under 5 minutes?
A flag is the panel's signal that something is off and the session should be re-reviewed by a second panelist. Flags are not the same as low scores; they are uncertainty calls. Three signals account for most under-five-minute flags. The first is unexplained acceptance — a major change in the diff that has no decision-point annotation and no defence audio touching it. The reviewer cannot tell whether the engineer chose it or accepted it without reading. That uncertainty is enough to flag.
The second is contradiction under defence — the engineer's annotation says one thing about the trade-off and their audio defence says another. The reviewer is not looking for hesitation; hesitation is fine. They are looking for the engineer to be internally consistent about what they decided and why. Two contradicting statements is a flag every time.
The third is silent regression-risk — a change that crosses a boundary (auth, schema, external API) without any test, comment, or defence mentioning the boundary. The reviewer is not deciding whether the change is wrong; they are deciding whether the engineer noticed the boundary. Silent boundaries get flagged because they cannot be scored fairly without a second opinion.
The donut below shows where reviewer time actually goes in a twelve-minute Hard-tier review. Decision points and defence audio combine to two-thirds of the budget. The diff spot-check is small because the diff is not the evidence — the replay is.
Why does the replay matter more than the diff?
The diff tells the reviewer what changed. The replay tells the reviewer why. In a world where the model can write a clean diff against a wrong understanding, the why is the only thing that distinguishes a defensible decision from a lucky one. The replay carries the reasoning that the diff cannot.
The table below lists the green flags and red flags reviewers actually call out in their session notes. Severity is the running cost to the final score, on a 1–5 scale, where 5 is "this is sufficient to fail the session on its own." Green flags are not the inverse of red flags; some red flags have no green-flag counterpart, because the absence of the signal is itself a signal.
| Signal | What it tells the reviewer | Common cause | Severity |
|---|---|---|---|
| Red: unexplained acceptance | The engineer cannot reconstruct the decision. | Accepting an AI suggestion without naming the trade-off. | 4 |
| Red: contradiction under defence | The engineer did not internalise the choice. | Writing the annotation before the audio defence; not reconciling. | 5 |
| Red: silent boundary crossing | The engineer did not notice the boundary. | Accepting changes to auth, schema, or external calls without comment. | 5 |
| Red: heavy polish, light verification | The engineer optimised before verifying. | Optimize-first iteration habit; cleanup before edge-case checks. | 3 |
| Green: position change under pushback | The engineer updates when evidence is good. | Treating the defence as a conversation, not a defence. | — |
| Green: named alternative considered | The engineer chose, did not default. | Annotation lists two options and the discriminating criterion. | — |
| Green: boundary called out in advance | The engineer noticed; reviewer does not have to. | Explicit comment or test at the auth or schema line. | — |
What separates a 4.0 from a 4.5 on defensibility?
The 4.0 and the 4.5 both pass. The difference is narrow and it is in one place: how the engineer handles a point the reviewer raises during the defence audio that the engineer had not considered. A 4.0 engineer holds the position and explains it again. A 4.5 engineer updates the position when the new point is good, and explains why it is good. The 4.5 is rewarded not for changing their mind but for changing it for a clearly-articulated reason.
This is the single most under-prepared-for behaviour in engineers approaching their first Hard-tier session. Most engineers practice defending; few practice updating. The simple drill: take a decision you defended last week, ask a peer to push back hard on it, and notice whether you can update without losing footing. Five rounds of that drill is the cheapest path from a 4.0 to a 4.5.
Why does the read flow stay fixed across reviewers?
The natural question is whether reviewers actually follow the read flow or improvise. We tracked the answer during the v2.6 calibration window. Most reviewers — 38 of 46 — follow the flow to within a minute on each stage. The other eight deviate, and the deviations group into two clusters. The first cluster reads diffs before decision points; the second cluster scores while reading instead of at the end. Both clusters show measurably lower inter-rater agreement on the defensibility dimension. Neither cluster passes calibration on the next quarterly review without coaching.
The flow is fixed because we have measured what happens when it is not. Reviewers who read the diff first form a code-quality hypothesis that crowds out the reasoning hypothesis; their defensibility scores correlate too tightly with correctness scores, which is the opposite of what we want. Reviewers who score as they read run out of attention by the spot-check stage; their final scores skew toward the dimensions they scored early. The flow is not a productivity hack. It is a fairness mechanism.
The downstream consequence for engineers is small but useful. You can predict which dimension the reviewer is reading for at any minute of the session. The first three minutes are about your framing and your decision points; that is where the defensibility score is forming. The next four minutes are about your defence audio; that is where AI-rejection sense and replay clarity are forming. The last five minutes are about the diff and the final scoring; that is where correctness and regression cost are forming. An engineer who knows the order knows where to invest preparation time. The first three minutes deserve a third of the preparation, not a fifth.
The reviewer-calibration study with inter-rater data and reviewer cluster analysis.
From Codritium Research
Panel Reviewer Calibration
What makes a panel reviewer reliable. Inter-rater dynamics, drift over weeks, the warm-up effect, and the strictness/agreement tradeoff that emerged once we ran 240 calibration sessions.
Common questions
Are panel reviewers paid?
Yes. Panel reviewers are working engineers contracted at a per-review rate calibrated against the median senior hourly. They are not Codritium employees, and they rotate quarterly to prevent stylistic drift. The full calibration protocol is in the linked research piece.
How long does a Hard-tier review take?
Twelve minutes median, sixteen minutes for sessions that get flagged for second review. The timeline in this post is the median flow. The hardest sessions to score are the borderline ones where the reviewer spends extra time on the defence audio.
Can you appeal a score?
Yes. An appeal triggers a third-reviewer pass with the original two reviewers' notes redacted. The third reviewer cannot see the original score until after they score independently. Appeals overturn the original outcome in about 9% of cases — most appeals are denied, but the rate is non-zero by design.
Do reviewers talk to each other before scoring?
Not for individual sessions. Reviewers calibrate as a group quarterly on a fixed practice set; they do not discuss live sessions before scoring. Inter-rater agreement is measured across the practice set and reported with the rubric version.
What's the reviewer agreement number?
0.81 on the Hard tier, weighted across the six rubric dimensions. The dimension with the highest agreement is correctness (0.91); the lowest is defensibility (0.74). Agreement on defensibility rose from 0.62 in 2024 — most of the gain came from tightening the rubric anchors, not from changing the reviewers.
Where to go next
- The calibration study: Panel reviewer calibration
- The rubric these reviewers score against: The defensibility rubric, 2026
- The skill the read flow is hunting for: Defensibility, the engineering skill AI can't replace