Across 16 weeks of cohort data, junior engineers paired with AI shipped 2.1 times the regression rate of the same engineers working solo on equivalent tasks. Senior engineers paired with AI shipped roughly the same rate as their solo baseline. The gap is real, it is structural, and it closes faster than most teams expect — but only with a specific kind of coaching. This piece walks through what the data shows, why the gap exists, and what closes it.
Why this matters
The conversation about AI and junior engineers is usually framed as "are juniors at risk." The data points to a sharper question: what specifically degrades when an unverified suggestion is shipped, and how do you teach the verification habit fast. The cost of the gap is paid by on-call rotations, by mentors, and by the juniors themselves when their PRs are reverted. The cost of closing it is eight to sixteen weeks of weekly practice. The case for investing in that practice — at the team level, not the individual level — is the case this piece is making.
What does the data actually show?
The chart below tracks regression rate over 16 weeks for three skill bands on the platform: top-decile, median, and bottom-tercile engineers. Regression rate is the share of merged PRs that produced a defect within four weeks of merge. We control for task category, model used, and reviewer.
Three things are worth noticing. Top-decile engineers barely move; they entered the study with a high verification habit and there is little to close. The median band shows steady improvement throughout. The bottom-tercile band — which includes most junior engineers in the study — starts well above the median and closes about 56% of its gap to the median by week 8, and most of the rest by week 16.
The picture changes when you cut by task category. Some categories show a much wider gap than others, and the categories where the gap is widest are exactly the categories where verification is hardest to fake.
| Debugging | Security | Refactor | Feature build | Migration | |
|---|---|---|---|---|---|
| Top decile | 3.6% | 4.1% | 3.0% | 3.2% | 3.5% |
| Median | 5.8% | 7.2% | 4.9% | 4.6% | 6.1% |
| Bottom tercile | 11.8% | 14.2% | 8.6% | 7.1% | 12.0% |
Debugging and security show the biggest gap because they are the categories where the suggested fix often looks like the real fix but isn't. The bottom-tercile regression rate on security tasks is more than 3x the top-decile rate. Refactor and feature build are closer because the cost of a wrong suggestion is more visible in the diff.
Why does the gap exist — what's the mechanism?
The mechanism is not raw skill. It is verification habit. We watched session recordings of 60 engineers across the bands working on the same debugging task. The behavioural difference was almost entirely about what happened in the seconds after a suggestion appeared.
A top-decile engineer reads the suggestion, hovers on the symbols they do not recognise, runs a smaller test than the one the model wrote, watches the test fail, and only then accepts. The accept-to-edit ratio is about 60/40. A bottom-tercile engineer reads the suggestion, accepts it, runs the full test suite, sees green, and moves on. The accept-to-edit ratio is about 90/10.
The funnel below tracks where AI-introduced bugs survive. The drop-offs at each stage are the points where the bug could have been caught and was not. The largest drop in the bottom tercile is at the verification stage; the largest drop in the top decile is at the review stage. The bottom-tercile engineer never gives the bug a chance to be caught locally; it ships and is found in CI or production.
The funnel narrows fastest at the second stage. That is the verification stage. That is where the coaching pays off.
What closes the gap fastest?
Three habits, in order of measured effect: paired verification, replay reviews, and rejection practice.
Paired verification is the simplest. The engineer pairs with a peer twice per week for a one-hour session in which they read each other's accepted suggestions and try to find a counterexample. Not the model's tests. Their own counterexamples. This single habit closes about half the bottom-tercile gap by week 8.
Replay reviews are the second. The engineer records a five-minute audio (or written) replay of one accepted suggestion per week, explaining why it was accepted in their own words. The recording is not for anyone else. It is for the engineer to hear themselves talk past the assumption. Most of the rest of the gap closes here.
Rejection practice is the third. Once a week, the engineer takes a suggestion they would normally accept and rejects it. Then they write the same code by hand. The point is not the code; the point is to relearn what the model has been doing for them. This is the practice top-decile engineers do without naming it.
None of these habits requires the engineer to use AI less. Telling juniors to use AI less is the response that does not work — they will use AI, and they should, because the floor on first-draft quality is meaningfully higher with AI. The habits above raise the verification floor to match.
Should juniors stop using AI?
No. The data does not support that response, and the structural argument against it is straightforward. AI use is now the default in most teams. A junior who avoids AI is solving a different problem than the one their team is solving. The right response is to keep AI in the workflow and add the verification habit. The cohorts that did this closed most of the gap in eight weeks. The cohorts that were told to use AI less did not close the gap and reported lower job satisfaction.
The honest framing for a junior is: AI raises the floor on what you can ship today, and lowers the floor on what you understand about what you shipped. Both are true. The verification habit closes the second floor without lowering the first.
The full study with band × category breakdown and 16-week intervention data.
From Codritium Research
Regression Rate by Skill Band
How often a 'fix' introduces a new bug, segmented across Easy / Medium / Hard sessions and twelve task categories. The fix-then-regress feedback loop, charted.
Common questions
Are these numbers tool-specific?
Across our sample, the gap between bands was larger than the gap between tools. The same band shipped within one percentage point regardless of which mainstream model was the primary pair. The mechanism is the verification habit, not the model.
Does seniority just mean more years?
No. Skill band is measured by rubric score on a calibrated set of platform challenges, not by tenure. About 18% of bottom-tercile engineers in the study had more than five years of experience.
What's the single best habit to adopt?
Paired verification, twice per week, sixty minutes per session. It is the habit with the largest measured effect on regression rate in the shortest time.
How fast does the gap close in practice?
About 56% of the bottom-tercile gap closes by week 8 with weekly practice. About 85% closes by week 16. The remainder is not noise — it reflects engineers who pick up the habit but do not internalise it; the fix there is mentoring, not more sessions.
Are top-decile engineers also faster?
No. Top-decile engineers are slightly slower per task. The total cost per merged-and-stable change is lower because they ship fewer regressions. The framing 'AI makes seniors faster' is a misread of the data — it makes them less wasteful, which is different.
Where to go next
- The full study: Regression rate by skill band
- The bug taxonomy underneath the funnel: Where AI pair programming fails
- The skill that the verification habit feeds into: Defensibility, the engineering skill AI can't replace