Engineers, hiring panels, and tool vendors are now using the same words to mean different things. A "rejection rate" on one team is the share of AI suggestions the engineer declined. On another team it is the share of AI-written PRs that were reverted in production. Both are useful numbers. They are not the same number. This glossary is the one we hand new reviewers, new candidates, and new platform partners so a meeting can start on the same page.
Why we wrote this
We watched four teams hire from the same candidate pool last quarter and reach opposite verdicts on the same engineer. The reason was vocabulary. One team's "AI-assisted" was the other team's "AI-autonomous." One team scored "defensibility" by replay clarity; another scored it by retro post-mortems. Engineers cannot signal what they cannot name, and reviewers cannot calibrate what they cannot name either. A shared glossary is not a style choice. It is the bottom layer of the hiring stack and the review stack. Most of these terms predate 2026; what is new is that they now appear in the rubric, the JD, and the on-call doc at the same time.
How the terms cluster
Roughly half the glossary names a skill or a habit an engineer practices — defensibility, in-the-room judgment, AI-rejection, vibe coding. Another quarter names a workflow primitive — replay, prompt pattern, session, panel review. A smaller cluster names a unit of measurement — regression rate, composite score, inter-rater agreement, Krippendorff's α. The last cluster names a process artefact a team produces — rubric, sample-size badge, worked example, replay annotation. Mapping each term to a family helps reviewers see which dimension a candidate is actually being scored on, and which dimension is missing entirely.
Which terms are new in 2026
Eight terms in this glossary did not exist in industry usage before 2024: AI-rejection rate, auth boundary drift, cohort drift, defensibility (in its rubric-bearing form), plausible nonsense, prompt pattern, replay annotation, and vibe coding. A handful — rubric, panel review, regression rate — kept their old shape but acquired AI-specific subtypes. The rest are inherited from earlier interview and review practice, sharpened to fit the new artefacts. The table below shows the five terms our review team flagged as most often misused in 2026 PR descriptions, JDs, and retro docs.
| Term | Common misuse | What it actually means |
|---|---|---|
| AI-rejection rate | Share of suggestions a tool's gate rejected. | Share of the engineer's accepted suggestions that survived the reviewer, post-hoc. |
| Defensibility | How confidently you can explain a choice on the spot. | Whether a reviewer who was not in the room can reconstruct the choice from the artefacts. |
| Replay | A video of the IDE session. | The structured record of decisions, prompts, and revisions tied to the diff. |
| Vibe coding | Any AI-assisted coding session. | A specific workflow where the engineer accepts most suggestions without explicit verification. |
| Composite score | An average of the rubric dimensions. | A weighted score where defensibility and regression carry more than correctness at the senior bar. |
Below is the full glossary. The three terms with their own callouts are the ones our reviewers ask about most.
- AI-rejection rate
- Share of an engineer's accepted AI suggestions that a competent reviewer later rejected or revised. Measured per PR, averaged per quarter.
Example: Reviewer rejects 4 of 20 accepted suggestions across a sprint; rejection rate is 20%. - Auth boundary drift
- A failure mode where AI-generated code silently widens or narrows the authorization scope of an existing handler. One of the top three bugs in our 2026 corpus.
Example: A new endpoint inherits a tenant filter from the file but loses the per-row check. - Cohort drift
- A measurable change in the median score of a cohort across consecutive practice quarters, after holding the rubric and tier constant.
Example: A cohort's regression rate trends up three quarters in a row — a drift, not noise. - Composite score
- The weighted aggregate of the six rubric dimensions. Defensibility and regression cost carry the most weight at the senior bar.
Example: A candidate with 9/10 on correctness but 4/10 on defensibility composites to a mid-band score. - Defensibility
- The ability to reconstruct, justify, and revise a code decision in front of a reviewer who was not in the room. The rubric dimension most predictive of hire outcomes in 2026.
Example: The engineer explains the retry budget by tying it to the SLO, not to a model suggestion. - Easy-to-Hard band
- The challenge-tier scale we score against — Easy, Medium, Hard. Each band has its own rubric weights and reviewer expectations.
Example: A defensibility score of 7 means different things on Easy and Hard. - Feedback loop
- The shortest cycle between an engineer's decision and a graded signal back. In AI-assisted work, shorter loops correlate with lower regression rates.
Example: A pre-commit test gate is a tighter feedback loop than a weekly retro. - Hallucination
- Output that is syntactically plausible but semantically wrong — an API that doesn't exist, a field that was renamed, a behaviour that was never specified.
Example: Code that imports `crypto.randomUUIDv7` because the model averaged across versions. - Hard-tier
- The senior-bar challenge tier on Codritium. Hard-tier work includes panel review, replay annotation, and defensibility scoring.
Example: A Hard-tier challenge expects a written defence of every non-trivial choice. - In-the-room
- A property of judgment exercised live, while the model is offering options and the engineer is deciding. Distinct from after-the-fact defence.
Example: An in-the-room rejection is faster and cheaper than a reviewer rejection. - Inter-rater agreement
- The degree to which two or more reviewers, scoring the same artefact independently, arrive at the same numeric score.
Example: Codritium Hard-tier panels currently sit at α = 0.81 for defensibility. - Krippendorff's α
- A statistical measure of inter-rater agreement that handles missing data and multiple raters better than Cohen's κ. The metric we use for rubric calibration.
Example: α above 0.80 is the threshold for a rubric dimension being considered stable. - Loop (interview loop)
- The sequence of interview stages a candidate moves through. In 2026 most senior loops include at least one replay-graded round.
Example: A four-stage loop with a take-home, a system design, a replay defence, and a panel review. - Panel review
- A multi-reviewer grading session in which two or more calibrated reviewers score the same submission and reconcile their scores.
Example: Hard-tier challenges go through a two-reviewer panel before a verdict is released. - Plausible nonsense
- AI output that reads like a competent answer but does not survive contact with the actual system. Distinct from hallucination because the shape is right and only the content is wrong.
Example: A retry block with the right structure but a backoff that exceeds the upstream timeout. - Prompt pattern
- A reusable prompt shape with a named purpose — a recipe an engineer applies repeatedly across tasks. Patterns travel between tools; prompts don't.
Example: The 'unit-test-first' pattern asks the model to draft tests before the implementation. - Pull-request review
- The synchronous or asynchronous reading of a PR by a non-author engineer, ending in approval, change requests, or rejection. The most common feedback loop in industry.
Example: A two-reviewer rule for production paths, one-reviewer for internal tools. - Rejection event
- A single instance of an accepted AI suggestion being rejected or revised by a downstream reviewer. The unit of measurement behind rejection rate.
Example: Reviewer comments 'use the existing helper'; that's one rejection event. - Regression rate
- Share of merged PRs that introduce a defect the test suite later catches or production later reports. Measured per author, per quarter.
Example: A 6% regression rate is around the median for senior engineers in 2026. - Replay
- The structured record of a coding session — prompts, decisions, false starts, and the resulting diff — produced alongside a merged PR.
Example: A replay is what a panel reviewer reads after the diff. - Replay annotation
- An engineer's written notes inside a replay that point a reviewer to the load-bearing decisions. The shortest path to a high defensibility score.
Example: An annotation that says 'rejected suggestion at L42 because the helper already exists'. - Rubric
- The structured scoring sheet used by reviewers. The Codritium rubric has six weighted dimensions and a documented inter-rater agreement target.
Example: Rubric v2.6 weights defensibility at 0.28 and correctness at 0.18. - Sample-size badge
- A visible annotation on a published statistic that names n, the cohort, and the date range. Numbers without one should be read sceptically.
Example: (n=412, Q1–Q2 2026) - Session
- A bounded period of work on a single challenge or task, with a defined start and end. The base unit for replay capture.
Example: A 90-minute session ends when the candidate submits or the timer runs out. - Spike
- A short, exploratory coding run aimed at answering a single question, not at shipping code. The output is usually thrown away.
Example: A two-hour spike to check whether the new client library handles retries the way we need. - Stale context
- AI output produced against an outdated mental model of the codebase — using deleted helpers, old field names, or removed endpoints.
Example: Code that references a `legacyAuthClient` deprecated three releases ago. - Take-home
- An asynchronous coding assignment a candidate completes off-platform, on their own clock, then defends in a follow-up session.
Example: A 4-hour take-home plus a 45-minute replay defence. - Top-tercile
- The top third of a measured distribution. The shorthand we use when a cohort's headline number hides a long tail.
Example: Top-tercile candidates show a 38% lower regression rate than the median. - Vibe coding
- A workflow in which the engineer accepts most AI suggestions without explicit verification, optimizing for flow rather than for defensibility. A 2024 coinage that has stabilized into a named anti-pattern at the senior bar.
Example: Accepting six suggestions in a row, then merging without re-reading the diff. - Worked example
- A complete, annotated walkthrough of a single problem from setup to merged PR. The most efficient training artefact when teaching defensibility.
Example: A 12-step replay of a debugging session, annotated at every decision.
The full v2.6 rubric, the weighting of each dimension, and the inter-rater agreement numbers that anchor most of the measurement terms above.
From Codritium Research
The Defensibility Rubric, 2026
The public scoring rubric behind every Codritium benchmark and certification: six weighted dimensions, twelve task categories, inter-rater agreement, and a frozen reference quarter.
Common questions
Are these terms specific to Codritium?
About a third are. Defensibility, replay, AI-rejection rate, and composite score all carry our specific rubric meanings. The rest are common across senior reviewers in 2026, with small wording variants.
Will these terms change next year?
Some will. Vibe coding and plausible nonsense are still settling. Defensibility and replay look stable — the rubric they live in is on its sixth revision and the deltas are now small.
Is there a canonical source?
Not industry-wide. The closest things are the rubric we publish at /research/benchmarks/defensibility-rubric-2026 and the panel-reviewer calibration notes. Inside a team, the canonical source should be your own JD plus rubric — pick one and link the rest to it.
Which term is most often misused?
AI-rejection rate. It is the only number that requires a reviewer in the loop, and teams without one usually substitute a tool-side metric that measures something different.
Are some of these gatekeeping jargon?
Two are at risk — composite score and Krippendorff's α. Both have plain-English equivalents. We keep them because the math behind them changes how you should read the published numbers.
Can I copy this glossary into my company wiki?
Yes. Attribute it, fork it, and edit the entries to match your rubric. A glossary that disagrees with your rubric is worse than no glossary.
Where to go next
- Read the rubric the measurement terms come from: The Defensibility Rubric, 2026
- See the reviewer calibration data: Panel reviewer calibration
- Practice replays in the Codritium challenge catalog
- Background reading: Defensibility: the engineering skill AI can't replace