If AI Is Allowed in Your Interview, LeetCode Measures Nothing

If AI is allowed in your interview and you keep a LeetCode round, you've measured nothing about the person in front of you. Here's the short answer for how hiring should change: kill the algorithm round, and score two things a model can't do for the candidate — whether they catch a broken or underspecified requirement, and whether they reject plausible-but-wrong AI output instead of shipping it.

First, let's pin down who we're arguing about. By "AI-native talent" I mean an engineer who learned to build with a coding agent in the loop from day one — someone whose default workflow is prompt, read the diff, decide whether to trust it — rather than someone who learned to code first and bolted AI on later. That's the population everyone has opinions about, so it's worth naming before we go further.

Here's the mechanism the whole argument turns on. A coding agent solves any published algorithm problem instantly and idiomatically. So the algorithm score now correlates with the candidate's model, not the candidate. The fix isn't a new tool policy. It's new tasks whose difficulty scales with the person rather than the tool.

There's a claim in circulation that some companies' most effective engineers are young and "AI-pilled." A version of it is usually traced to remarks by Clay Bavor (co-founder of Sierra, formerly of Google) — but I can't hand you a clean, verifiable citation for the exact wording, so treat the attribution as approximate and the claim as a paraphrase, not a sourced quote. Even where it's true, it tends to come from small, mostly-greenfield teams. What travels is the reasoning underneath it, not the headcount.

The signal didn't degrade. It inverted.

A coding interview is a signal-extraction problem. You get 60 to 90 minutes to predict 6 to 24 months of work. The whiteboard round worked, weakly, because raw problem decomposition was expensive to fake, so it served as a proxy for junior productivity.

AI collapsed the cost of faking that proxy to near zero. So the signal-to-noise ratio didn't just drop. It inverted. A candidate with a good model now looks stronger than a better candidate with a worse model. You're ranking model providers and calling it hiring.

That's the whole case for removing the round, not merely allowing AI on it. If AI is permitted, the round's correlation to job performance goes to zero, and a zero-signal round only adds noise.

What actually survives AI in the room

Once the tool is allowed, only two kinds of difficulty can't be outsourced to it.

The first is task ambiguity. The requirements themselves are wrong or underspecified. Agents optimize against the spec they're handed. They don't flag that the spec contradicts itself. Give a candidate a partially-broken specification and the signal is whether they notice "this API contract can't be satisfied as written." No model surfaces that for them.

The second is verification burden. The candidate has to reject plausible output. This is the important one, because LLMs are trained to produce high-likelihood continuations. The failure mode isn't gibberish. It's confident, well-formatted, subtly-wrong code: a hallucinated endpoint, a re-introduced N+1 query, a duplicated service abstraction. Catching it requires holding an independent model of correctness and acting on distrust.

Speed is not the differentiator here. The best AI-native juniors don't type faster. They know when to distrust the model and write a test to pin it. Call it calibrated distrust. That's the thing you're hiring for.

Why AI code creates debt unit tests can't catch

This is the part the "raw fluency is fine" crowd underrates, and it's a clean causal chain, not a buzzword.

An LLM's context window is finite and doesn't contain your dependency graph. So the model reasons locally. It satisfies the immediate ticket. A mature codebase, meanwhile, encodes global invariants that live between files and aren't visible in any single one: the ORM layer that exists specifically to kill N+1 queries, the transaction boundary that makes a cache-invalidation race safe, the one service abstraction everything routes through.

The agent can't see why those exist. So it produces a diff that passes all unit tests, because tests assert local behavior. Violates a global invariant, re-introducing the N+1 or forking the abstraction. And ships, because review velocity is up and the diff looks clean.

The debt lands specifically in cross-cutting concerns: auth, transactions, cache invalidation. Those are exactly the invariants a local test can't pin. Imagine a mature codebase — say a 2M-line monorepo — old enough to have load-bearing invariants like these. In a place like that, you'd expect AI-heavy PRs to revert at a noticeably higher rate than the baseline, and to see the reverts cluster in exactly those cross-cutting layers, because that's where the local-reasoning failure mode has something to break. That's a prediction from the mechanism, not a measured number. But it's the first number worth collecting.

This is also why the failure is invisible at interview time by construction. You'd need a codebase old enough to have load-bearing invariants to even reproduce it. In a greenfield product, no AI-generated PR has detonated yet. That's the survivorship bias inside the "our 22-year-olds are our best engineers" story: you're selecting from people already hired at a top startup, and grading them in a codebase too young to expose the failure mode.

The AI-native delta isn't cheating. It's exposure.

Here's where the senior-engineer contempt goes wrong. The strongest AI-native workflow treats the model as an unreliable pair programmer, not an oracle. That flips the work to context-engineering first: curate which files and docs enter the window, write a spec the agent reads before it codes, keep a CLAUDE.md of repo invariants so the model's local reasoning is fenced in by global rules. That's a genuinely different, and in some ways more disciplined, mental model than a senior who just starts typing.

The real gap is downstream. Because the agent always hands back a plausible explanation, the AI-native never has to earn the systems intuition that Stack Overflow-era engineers ground out through painful debugging. So the weakness is specific and predictable: debugging non-reproducible bugs, and reasoning about behavior under production load. Not "they're shallow." They got their systems knowledge in a market where you can buy a confident wrong answer for free, and a confident wrong answer teaches you nothing.

Which is also why "just prompt better" fails. Without an internal system model, better prompts produce more confident wrong answers, not fewer. A strong candidate will already know this about themselves — and that self-knowledge is something you can probe for in the room.

Rounds that actually discriminate

Replace the algorithm round with live work samples designed around the two AI-resistant properties.

The flaky-API round, about 90 minutes: ship a feature against a deliberately unreliable internal API using Cursor or Claude Code. Plant the trap — the agent hallucinates an endpoint that doesn't exist. You're scoring whether the candidate blindly integrates it or writes a test to pin the contract first.

The broken-spec round: hand a partially self-contradictory specification. The signal is whether they flag that the requirements can't all be true. Agents won't.

Then score intervention quality from a screen recording. Classify every agent interaction as blind-accept, edit, reject-with-a-stated-reason, or re-prompt-with-a-reason. That distribution correlates with on-the-job PR quality. Whiteboard scores didn't.

Score diff-review velocity as a first-class skill: how fast do they catch a subtly-wrong AI edit buried in a 400-line PR? And put eval-writing and prompt decomposition on the rubric, not in the "soft skills" bucket — breaking a vague ticket into verifiable sub-tasks is the job now.

Finally, a systems-intuition probe: before they touch existing code, ask them to explain why the current code is shaped the way it is. That surfaces whether they read for invariants or just for the diff.

One caution. None of this is a template you copy from another company. Signal design is a function of codebase maturity times team size. A loop that fits a small, greenfield team will underweight exactly the probes a 2M-line monorepo team needs weighted heaviest. Build for your own codebase's invariants, not someone else's blog post.

The fight nobody's won yet

Two disagreements are genuinely open, and pretending they're settled is how you build a bad loop.

Is systems intuition interviewable in 90 minutes at all? One camp says no: the failure mode is invisible at interview time by construction, and only shows up after six months in a legacy codebase. The other says the "explain why this code is shaped this way before you touch it" probe surfaces it live. Both are partly right, which is why you weight it rather than treat it as pass/fail.

And does any single company's redesign generalize? Mostly no. What generalizes is the principle: measure intervention quality, not tool policy. The specific rounds are yours to build.

So here's an honest question worth arguing about. If you had exactly one 90-minute round to predict whether a hire would ship debt into your auth layer, which of these would you run? I don't think there's a clean answer. Where would you bet?

FAQ

Does "we changed our interview" usually mean something real? Usually not. Most of the time it means "we now allow Copilot." Allowing AI while keeping LeetCode isn't an overhaul. It's a benchmark of whose model is bigger. A real overhaul is a new rubric that scores intervention quality.

Are AI-native juniors worse hires? No. The ones who fail fail on verification discipline, not systems knowledge, and verification discipline is testable at interview time. That's the whole point of the flaky-API round.

Should seniors be worried? Only the ones whose edge was typing speed and syntax recall. The edge that's appreciating is systems intuition plus calibrated distrust, and right now the AI-native has to manufacture the former while the senior often already has it.

Do these three things this week

Delete your algorithm round and replace it with one flaky-API work sample where the agent hallucinates an endpoint. Score whether the candidate pins the contract.

Start scoring intervention quality from a screen recording: tag each agent interaction as blind-accept, edit, reject, or re-prompt-with-a-reason.

Add one invariant probe to every loop touching a mature codebase: explain why this code is shaped this way before you change it.