Why this matters
Take-home assessments and live coding rounds measure different things. Take-homes let candidates think, look things up, and shape an answer over hours. Live rounds compress that into 45 minutes under a stranger's eyes. Hiring teams keep both because they don't trust either one alone.
The question we wanted to answer is narrower than "which is better." We wanted to know whether a take-home score from the platform — composite, blind-graded, with replay — predicts the outcome of a live round at a named employer. If it does, the take-home is doing work. If it doesn't, candidates and reviewers are spending hours each on a signal nobody uses.
We had the data to check. 412 candidates, 1,860 rounds, 23 employers across H2 2025 and Q1 2026. This piece walks through what we found, the model we built, where it lied, and the feature that fixed it.
Methodology in one paragraph
A "take-home" here is the candidate's blind-graded composite across at least three Hard-tier platform sessions in the 60 days before their first live round. A "live outcome" is the per-round verdict the employer's loop produced — Strong Hire, Hire, No Hire, Strong No Hire — collected with candidate consent. We restricted the sample to employers whose loops do not gate on the platform score directly; the 9 employers that do were excluded to avoid feedback contamination. The window is 2025-08-01 to 2026-03-31. Rounds with missing verdicts (4.1% of the total) were dropped. All correlations are Spearman ρ unless noted.
412
Candidates tracked
≥3 Hard-tier sessions before loop
0.58
Take-home → live ρ
Across all 23 employers
11.4%
False-negative share
Top-decile take-home, No-Hire live
0.07
Calibration error
Mean |predicted − observed| over deciles
The relationship is real, and noisy
A scatter is the right first picture. Each dot is a candidate. The x-axis is the take-home composite, scaled 0 to 100. The y-axis is the share of that candidate's live rounds that cleared to at least Hire. The relationship is positive and clearly monotonic. It is also wide.
ρ = 0.58 is a real signal but it is not a verdict. The shape we care about is the tails. Strong take-homes have a long right tail — most clear, some don't. Weak take-homes have a short ceiling — almost none clear at top-decile rates. The asymmetry shows up later in calibration. A ρ of 0.58 is in the same neighbourhood as our previous interview-fit study (ρ = 0.62) but lower because this version excludes the loops that gate on platform score, which would have inflated the number.
The middle of the cloud is dense and wide. Between composite 50 and 70, the live pass rate spans from roughly 0.15 to 0.62. Two candidates with the same take-home composite can have triple the difference in live outcomes. That dispersion is the part of the picture you should keep in mind every time someone quotes a single ρ at you. The score is informative. It is not a function.
Residuals — where the model lies
We built a regression model on this. Take-home composite as the only feature, predicting offer probability. The residuals — observed live pass rate minus model-predicted pass rate — should look roughly normal if the model is doing its job. They don't.
The shape is roughly normal at the centre. The left tail is heavier. There are more bad-surprise candidates — strong take-home, weak live — than the symmetric model expects. We dug into the 53 candidates in the left tail (residual < −0.2) and found a pattern: most had a high take-home composite but a low AI-rejection-rate subscore. They scored well in conditions where AI could carry them and stalled in live rounds where the interviewer made AI the wrong answer.
The right tail is thinner but not empty. There are 19 candidates whose take-home composite was below the cohort median and who cleared live rounds at a rate the model did not predict. We looked at this group too. Most of them had a defensibility subscore well above their composite. They scored modestly on raw correctness, but they articulated their reasoning cleanly when the panel asked. The live-round format rewards exactly that skill. The take-home composite, weighted toward correctness, under-rates it.
The outcome mix by take-home decile makes the same shape visible. The bottom deciles are dominated by No Hire and Strong No Hire. The top deciles are dominated by Hire and Strong Hire — but a stubborn slice of No Hire remains, and it doesn't shrink to zero.
The decision tree, walked
A linear model misses the structure. A small tree finds it. We grew a depth-3 tree on the same 412 candidates with the live outcome (Hire-or-better as positive) as the target. The first split was the one we expected. The second split was not.
The root split is defensibility — the panel's score on how clearly the candidate defends their decisions on replay. Above 4.0 out of 5, the candidate enters the right branch; below, they enter the left. On the right, the next question is AI-rejection rate. Candidates who score high on defensibility and reject AI suggestions at a rate above 0.45 cleared 79% of their live rounds. Candidates with high defensibility but low AI-rejection cleared 48% — still good, but the gap is large enough that AI-rejection earns its own split.
On the left branch — candidates whose defensibility is below 4.0 — correctness on Hard tier becomes the deciding question. With Hard-tier correctness above 0.7, they still clear 34% of rounds. Without it, only 11%. The tree's overall accuracy is 71% on a held-out 20% test set, up from the linear model's 64%.
Calibration: tail compression at the top
A model can be accurate on average and still miscalibrated at the tails. We binned candidates by their model-predicted offer rate and compared the prediction to the observed pass rate inside each bin.
The middle deciles track each other. The top decile diverges. The model thinks the strongest 10% will clear 86% of their rounds; they clear 72%. This is the same shape as the residuals. Strong take-home performance is not a guarantee of strong live performance, and the model that pretends it is will keep over-promising at the top.
Calibration matters more than ρ for the hiring decision. Telling a recruiter "this candidate has a 60% chance of clearing your loop" is only useful if the 60% is actually 60%. At the top, it isn't.
Employer-by-employer — heterogeneity matters
The pooled ρ of 0.58 hides large differences between employers. Some loops are tight reflections of the platform — they value the same things, score them the same way, and the take-home composite predicts cleanly. Others use coding rounds as a screen and weight everything on system design and behavioural — there the platform composite, which is mostly coding-shaped, predicts less.
| # | Employer | Rounds | ρ | Top-decile lift | Offer-rate trend |
|---|---|---|---|---|---|
| 1 | Big-Tech A | 184 | 0.71 | +38pt | |
| 2 | Big-Tech B | 168 | 0.66 | +33pt | |
| 3 | Big-Tech C | 142 | 0.62 | +29pt | |
| 4 | Payments A | 118 | 0.69 | +36pt | |
| 5 | Payments B | 96 | 0.58 | +27pt | |
| 6 | Cloud A | 91 | 0.61 | +28pt | |
| 7 | Cloud B | 87 | 0.43 | +18pt | |
| 8 | Infra A | 79 | 0.66 | +31pt | |
| 9 | Search A | 71 | 0.54 | +24pt | |
| 10 | Devtools A | 64 | 0.48 | +21pt | |
| 11 | AI-Lab A | 58 | 0.39 | +16pt | |
| 12 | Trading A | 51 | 0.36 | +14pt |
Two patterns stand out. The high-ρ employers (Big-Tech A, Payments A, Infra A) are the ones whose loops weight on-keyboard correctness and decision defense — exactly what the platform measures. The low-ρ employers (AI-Lab A, Trading A) weight domain knowledge and rapport. We don't measure those. The honest reading is not "the platform predicts everywhere." It is "the platform predicts well at loops that look like the platform, and worse at loops that don't."
The sparkline trends in the right column also tell a story. Most of the high-ρ employers show a rising offer-rate trend — the candidates clearing their loops are getting stronger over the six quarters we tracked. The lower-ρ employers show flatter trends. We do not have a clean reading of cause and effect here. One reasonable hypothesis is that high-ρ loops select the kinds of candidates who keep practicing in our format, which then improves those candidates' platform performance, which then improves future ρ in a self-reinforcing way. We are watching the trend lines but not making causal claims from them.
Feature ablation — AI-rejection is the surprise
We had built the first version of this model without AI-rejection rate. It is the subscore that captures how often a candidate declines an AI suggestion that would have shipped a regression. The first model used correctness, defensibility, regression rate, and spec fidelity. Adding AI-rejection rate moved the per-employer correlation more than any single change we'd seen.
The lift is sharpest at the loops that already correlated strongly — Big-Tech A, Payments A, Infra A — where adding AI-rejection pushed ρ above 0.65. It is essentially flat at Cloud B and small at Devtools A. The interpretation: where the loop tests "can you spot an AI mistake under pressure," the subscore that measures exactly that earns its keep. Where the loop tests anything else, it doesn't.
This is the part of the result we did not expect. We added AI-rejection rate as a control. It became the single best feature.
A small note on what AI-rejection rate actually measures. It is not "the candidate refuses to use AI." It is "across the candidate's sessions, when the AI proposes a change that would ship a regression (per the rubric's regression-cost dimension), what share did the candidate reject or modify before merging." A high AI-rejection rate is not skepticism. It is a habit of reading the AI's output carefully enough to catch its mistakes. Loops that test for that habit get more predictive lift from a metric that measures it. The mechanism is mundane once we name it.
Caveats
This is observational. Candidates self-selected into the platform, and into the consented-tracking program inside it. There is no random assignment of candidates to take-homes-only versus live-only loops; we can describe the joint distribution but not isolate the causal contribution of either format.
The live verdicts are themselves noisy. Inter-loop disagreement at our top employer is 18% — two loops on the same candidate reach different verdicts almost one time in five. We pooled across rounds to dampen this, but the noise floor on the outcome variable is real and bounds how high any ρ can climb.
We excluded employers whose loops gate on platform score because including them would inflate ρ through definitional feedback. That excluded subset has its own story, which we describe in the methodology appendix but not here.
Subscore values changed mid-Q1 2026 when we adopted rubric v2.6. We re-scored prior sessions retroactively using the new rubric so the composite numbers are consistent across the window. Engineers familiar with rubric versioning will notice some sessions had to be panel-reviewed twice.
What we'll do next
We re-run this study every six months. The next cut adds 11 employers and approximately 600 more rounds, plus a second outcome variable — accepted-offer rate, not just clear-to-offer rate, which is shaped more by candidate preference than employer judgment. The training and held-out folds are versioned in the open. We welcome independent replications, particularly from employer teams that want to fit a model on their own loop and compare it to ours. The rubric source is in the Defensibility Rubric.