The "First Draft" Test Is Half Right — Here's the Math It's Hiding

Let's start with the number, because everything else grows out of it.

An agent isn't one call to a model. It's a model in a loop — it picks a tool, reads the result, picks another tool, and does that over and over. So "did the task succeed?" isn't one dice roll. It's a chain of them. And chains do a nasty thing: they compound.

Say each step succeeds 95% of the time. Feels great. Now run 14 steps. The whole task only works if every step works, so you multiply: 0.95 to the 14th power. That's about 49%. Bump per-step reliability to 99% and the same 14-step task jumps to about 87%. Stretch the horizon to 40 steps at 95% and you crater to 13%.

One honest caveat before we build on this: those numbers assume each step's failure is independent of the others — a clean 0.95 multiplied 14 times. Real agents don't work exactly like that; one bad step often poisons the next, and some steps are far riskier than others. So treat the math as a simplification, not a measurement. It's here because it makes one thing vivid that nothing else does: horizon length, not raw model quality, is what quietly decides whether a task lands.

Sit with that for a second, because it flips the usual complaint on its head. People say "agents aren't ready because the model isn't 99% reliable." But look at what the numbers actually do — a four-point change in per-step reliability can double or halve whether the whole task lands, depending on how long the task is. So "is the model good enough?" is the wrong question. The variables that matter are per-step reliability and horizon length. End-to-end accuracy, by itself, tells you almost nothing you can act on.

Okay. So if the math is this brutal, why does anything work at all?

Here's the move, and it's genuinely elegant once you see it. Imagine the agent does 7 steps, produces something, and a human looks at it and okays it before it goes any further. What just happened, mathematically? The error chain reset to zero. You're no longer computing 0.95 to the 14th. You're computing two separate 7-step chains, each one much healthier, stitched together at a checkpoint.

That's the intuition behind "first draft" work — a framing Harrison Chase, founder of LangChain, laid out on Sequoia Capital's Training Data podcast. His point: the reason today's agents are usable isn't that they're smart enough to finish a job unattended — it's that a human sits at a review boundary and treats the output as a draft. Worth flagging: that's one expert's perspective, not settled industry consensus, and when Chase talks about "99% reliability" he's making a directional argument, not citing a benchmarked number. Take it as a lens, not a measurement.

Here's where we (LensUp) want to extend that framing rather than just repeat it — because the moment you take Chase's "draft" idea seriously, it stops being about drafts and starts being about math. What the review boundary really does is horizon segmentation — a term we're using to name the mechanism underneath: you're chopping one scary long chain into short segments where compounding hasn't had time to eat you alive. The human review boundary is the scissors. That reframe, and the ones that follow, are our editorial synthesis on top of Chase's starting point — not claims he made.

Which means the actual design question isn't the one everyone asks. It's not "is the output a reviewable draft?" It's: where do I place the cheapest checkpoint that resets the error chain?

And here's where the popular framing starts to leak.

The seductive version goes: "coding agents produce a pull request, a human reviews the PR, ship it — killer app." Sounds airtight. But watch what happens with a real example. An agent writes a PR that's 80% there. The remaining 20% is a subtle logic bug buried in a refactor. To catch it, the reviewer has to reconstruct what the agent was trying to do, trace the logic, and find the flaw — which can cost more than just writing the thing from scratch. Call that the draft tax (our term): an almost-done output where finishing it is more expensive than starting over.

So "produces a PR" is not automatically a win. It depends entirely on the shape of the errors. And the shape has two axes people constantly conflate: how easy the error is to fix, and how easy it is to detect. Report generation passes the test not because its errors are easy to fix, but because they're easy to spot — you read the report and the wrong number jumps out at you. A complex refactor fails because the error is expensive to detect, and detection is where the money goes. Same "reviewable draft" label, opposite outcome. (This detect-vs-fix split is us extending the draft idea, not something Chase spelled out.)

Now the part that quietly sinks more projects than any of this.

Even if you place a perfect review boundary, review can be pure theater. Here's why. In a plain single-model app — one prompt, some retrieved text, one answer — the context is code-determined. You know exactly what went in, because you wrote the prompt and you chose the chunks. In an agent, the context at step 14 is the emergent residue of the previous 13 steps, each of which could've pulled arbitrary stuff into the window. You literally cannot know what's in context at step N without replaying steps 1 through N-1.

So when a human "reviews the draft," what are they actually looking at? A conclusion with invisible premises. They can't see why the agent decided this. And if they can't reconstruct the why, they're rubber-stamping — the checkpoint stops resetting anything. This is exactly why traces are optional in single-model apps but non-negotiable in agents: the trace is the only artifact that reconstructs "what was actually in context when this decision got made." Reviewability isn't downstream of the output being text. It's downstream of controllable context.

Chain the whole thing together and it reads like this: controllable context → makes the trace reconstructable → makes the draft genuinely auditable → makes review a real error-reset instead of a rubber stamp → keeps each segment's reliability high → task succeeds. Snap any one link and "reviewable draft" is just marketing.

Now — one honest fork, because it's the disagreement that actually survives.

Everything above assumes latency is free. That's true for async knowledge work: writing, coding, reports. Nobody cares if the human reviews the draft an hour later. But drop this into an AI agent handling a production incident at 3am, and the calculus inverts. There, speed is the product. If you gate a pod restart behind human approval, congratulations — you built more expensive alerting. The review boundary there isn't about draft quality at all. It's about reversibility — and this reversibility distinction is another piece of our synthesis, the place where the draft framing stops fitting and needs a different primitive.

Reversible actions — restart a pod, scale a replica, roll back a feature flag — can run autonomously, as long as there's an automatic rollback rule. That rollback rule is the error-reset checkpoint; it just happens to be executed by a machine instead of a person. Irreversible actions — deleting data, changing DNS, editing permissions — surface as drafts for a human to approve. So it's the same primitive as human review — a cheap recovery point — but the unit you gate on is the action's reversibility, not the whole task.

So here's the test that replaces "is it a draft?" — four criteria we've assembled to make the selection concrete. A use case actually works when all four of these hold:

One — segmentable horizon: you can drop a recovery point that keeps each segment's reliability high. Instrument it, don't vibe it.

Two — auditable context: the reviewer can reconstruct why from a trace. Fail this, and you fail everything, regardless of output format.

Three — favorable error shape: mistakes are cheaply detectable, not just cheaply fixable.

Four — closing economics: review-plus-fix costs less than doing it yourself, and less than the cost of a wrong autonomous action.

Three things to do tonight:

① Open your agent's trace tool — LangSmith, Braintrust, whatever you've got — and measure per-step reliability, then multiply it out across your real horizon length. Stop looking only at task-level pass rates, and find the step that's actually failing.

② Take your top use case and score its error shape on two axes: detectability and fixability. If errors are expensive to detect — like subtle refactor bugs — flag it as a draft-tax risk and don't ship it as "produces a draft, done."

③ List every action your agent can take and split it into reversible vs. irreversible. Reversible ones get an automatic rollback rule and run on their own; irreversible ones become human-approval drafts.

One question I'm genuinely curious about: what's a use case you thought would work as first-draft work and didn't — and when you look back, was it the horizon, the error shape, or the context you couldn't reconstruct? That last one is the sneaky killer, and I'd bet it's more common than anyone admits.