Open vs Frontier Models: Why Distillation Is Closing the Gap

Short answer: open models are catching up to frontier models mostly by copying them, not by out-researching them. Clay Bavor, co-founder of Sierra, put the popular version of this bluntly — part of the gap, he says, comes down to "the willingness of Chinese companies to do scale distillation of the frontier models." If you can't build frontier yourself, you distill the ones who did.

That's roughly right and mostly misleading. Let me give you the actual mechanism, the one rule that decides where distillation works, and why "who wins" gets settled somewhere other than the model.

"Distillation" is the wrong word for what's happening

Real distillation — the technique introduced in Hinton, Vinyals, and Dean's 2015 paper "Distilling the Knowledge in a Neural Network" — trains a small student to match a big teacher's full output distribution. You minimize the divergence between their probabilities over the entire vocabulary. To do that you need the teacher's logits.

You don't get logits through a commercial API. At best you get top-k logprobs, usually just sampled tokens. So when people say a lab "distilled" a frontier model it only had API access to, they can't mean the real thing. What they actually did is large-scale supervised fine-tuning on the frontier model's completions, filtered by rejection sampling.

Keep that distinction, because it explains everything downstream. If you own the teacher, you can do white-box distillation on its internals. If you're copying a competitor you don't own, you're stuck doing black-box imitation on its outputs. US labs do the first with their own bigger models. Anyone copying a rival does the second.

The four-part pipeline

The black-box version has four moving parts, and each one explains a different piece of the gap.

Prompt distribution design. You decide what to ask the teacher — hard math, competitive-programming problems, tool-use traces, multi-turn agent flows. This is the real IP. Bad coverage means a student with holes exactly where you didn't think to ask.

Completion plus rejection sampling. You sample N answers from the teacher and keep the good ones. The filter is the whole game. For math and code you have a verifier: run the code, check the proof, compare the number. So best-of-N is high signal — you keep genuinely correct traces. For open-ended judgment there's no cheap verifier, so your filter is a weak reward model or an LLM acting as judge, and quality slides toward the teacher's average instead of its best.

Chain-of-thought distillation. When the teacher is a reasoning model, the sampled chain-of-thought is a search process written out as tokens. The student isn't copying the answer, it's copying the procedure. That's why distilling a reasoner is far more powerful than distilling a chat model — you're transferring a policy, not a lookup table. The teacher already paid the exploration cost, and you get the receipt.

Bootstrap, then run your own RL. After fine-tuning on distilled traces, you run reinforcement learning with verifiable rewards. GRPO — group-relative policy optimization, as described in DeepSeek's R1 paper — is the trick that makes this cheap: sample a group of answers, reward the correct ones, normalize within the group, so there's no separate learned reward model to train. Because RL explores beyond imitation, the student can find good traces the teacher never emitted.

The ceiling is real and fake at the same time

Here's the part most takes stop short of.

The "distillation ceiling" — the idea that a copy can't beat its original — is true for imitation-only work and for non-verifiable tasks. Product taste. Long-horizon planning. Customer-facing writing that has to read well. Agentic judgment. There's no verifier, so there's no reward signal, so there's no way to climb past the teacher. Hard cap.

It's different for verifiable tasks. DeepSeek's R1 paper showed a concrete result: distilling a large RL-trained teacher into a small dense model beat running RL on that small model directly. The teacher already did the expensive exploration; the student inherits it, then RL pushes further. Distilled open models have matched or beaten teacher and small-model baselines on specific math and coding benchmarks — in cases such as the DeepSeek-R1 distillations — but the results are uneven, benchmark-dependent, and don't prove they surpass the frontier teacher across the board.

So the ceiling depends entirely on whether the task has a cheap checker. That's the whole idea, and it deserves a handle: the gap is verifier-shaped. Open models reach parity first and fastest exactly where an answer can be auto-checked — because that's the only place the pipeline's filter works.

The uncomfortable corollary

Follow that one step past where it's comfortable. If open models close the gap fastest where there's a verifier, then the frontier premium isn't dying. It's concentrating on the tasks with no verifier.

Deal-risk assessment. Multi-step enterprise planning. Anything where "good" can't be scored by a script and a wrong answer is expensive. Benchmark parity on MATH quietly hides a widening gap on exactly the high-value agent work. The frontier keeps the slice that can't be distilled, because it can't be filtered.

It's not willingness, it's arithmetic

Bavor frames this as a cultural trait — the "willingness" of certain companies. That reads as national character. It isn't.

Two things make distillation rational for anyone, anywhere. First, cost. The contrast is order-of-magnitude, not exact: public estimates put frontier pretraining runs in the tens to hundreds of millions of dollars, while generating a distillation dataset through API calls is far cheaper — on the order of a small fraction of that, depending on volume and pricing. That's a cost-of-capital arbitrage, not a research achievement. Second, legal: OpenAI's and Anthropic's terms of service forbid using their outputs to train competitors, and cross-jurisdiction enforcement of those terms remains limited, uncertain, and largely untested. The "willingness" in the quote is legal-risk tolerance, not a technical or cultural edge.

And here's the part the framing leaves out: everyone distills. Every small production model shipped by frontier labs is distilled from a bigger internal model. This is the standard industry pipeline, not a foreign strategy.

You can prove who copied whom

Distillation leaves fingerprints. A copied model will often self-identify as the teacher ("I'm ChatGPT") when you don't prompt carefully. It inherits the teacher's refusal phrasing. Its token distribution carries statistical tells. It can show the teacher's eval contamination.

So "intent to distill" is both detectable in the artifact and beside the point. The interesting question was never whether someone distilled. It's what the copy can and can't do.

Who wins is decided at token economics

Now zoom out from the model, because "open vs frontier, who wins" is the wrong altitude. It's decided at cost per query.

In a production agent stack, most subtasks are boring: routing, extraction, formatting a tool call, reranking results. Call it the commodity 80%. A distilled 7–32B open model, self-hosted, wins on total cost of ownership for that 80% — and you can fine-tune it on your own domain data, which no closed API lets you do.

Distillation doesn't need to beat frontier to matter. It needs to be good enough to demote frontier to a fallback for the hard 20%. That's not a capability defeat. It's a revenue-per-query collapse. The frontier premium survives only on the shrinking slice where there's no verifier and an error is catastrophic.

The one thing worth arguing about

Two honest reads pull apart here.

One: the router architecture — cheap open model for the 80%, frontier for the 20% — erodes the frontier's economics until the premium barely holds.

The other: benchmark parity masks the widening judgment gap, so the premium doesn't erode, it concentrates onto the no-verifier tasks and stays fat.

Both can't be fully true. Which way it breaks depends on how much of real enterprise value lives in tasks you can't auto-check. If you run agents in production, that's the number I'd genuinely like to hear from you: in your stack, what share of the work actually has a verifier — and what share is pure judgment?

Do these three things tonight

• Stop saying "distillation" and name the regime. Open a doc and write down, for whatever model you're evaluating: black-box (API-output SFT) or white-box (you own the teacher). It changes what the model can possibly do.
• Audit your tasks for verifiers. List your agent's subtasks in a spreadsheet and mark each one "auto-checkable: yes/no." The "yes" column is where a distilled open model will reach parity — route those there first.
• Test the router math on one task. Take your highest-volume verifiable subtask, run a self-hosted distilled 7–32B open model against your frontier API on the same inputs, and compare cost per 1,000 calls. That single number tells you where your token budget is leaking.