"Nobody Knows Anything" — What Kent Beck Got Right, and What His Advice Hides

That's Kent Beck — the man who put the "test-first" in test-driven development, creator of Extreme Programming, and one of the signatories of the Agile Manifesto. On The Pragmatic Engineer Podcast episode "TDD, AI agents and coding with Kent Beck," he said the thing that's now being quoted everywhere: he's "trying all of the tools because right now, nobody knows what process is going to work best. Nobody knows anything. We should all be trying all the things that we can imagine. And then the truth will emerge out of all of that."

Everything Beck-specific below — the quote above, the delete-the-assertion move, the unplug-you line, the Smalltalk parser anecdote, the fast test suite — comes from that same episode. It's a generous, exciting stance. It is also half true, in a specific and checkable way. Let's find the seam.

Start with why "try everything" even feels rational. It feels rational because most people cannot see what varies between these tools. They carry the mental model that a better model means a better coding tool. That model is wrong, and its wrongness is exactly what makes the field look like chaos. The variance between Cursor, Claude Code, Aider, and Copilot Workspace does not live mostly in the model. It lives in three stages of a pipeline. Walk the pipeline and the chaos resolves into structure.

Stage one is context selection: deciding what the model even sees. Your repository is far larger than any context window, so the tool must retrieve and rank. Aider parses your files with tree-sitter, extracts the symbols — functions, classes, signatures — and builds a repo-map that ranks references with PageRank-like tag weighting, then packs the highest-signal symbols into a token budget. The model sees a skeleton of the whole repo. Cursor indexes the codebase and retrieves relevant context by embedding search, plus whatever files you've opened or @-mentioned. Claude Code leans the other way — it issues grep and read calls and builds context on demand, iteratively, rather than pre-packing everything up front.

From that, you can derive something instead of guessing it. On a large polyglot repo, whichever tool surfaces the right symbol wins before the model writes a single line. On a small, single-file repo, context selection is trivial — it's free — so the tools converge and the whole stage stops mattering. That is why the "best tool" flips by repo size. It isn't random. It's a term that dominates when the repo is big and vanishes when the repo is small. In the episode, Beck describes hacking on a Smalltalk parser — a tiny, self-contained thing. His context stage was free the entire time.

Stage two is edit application: getting a proposed change onto disk. Three strategies, three distinct failure modes. Whole-file rewrite — the model re-emits the entire file — fails two ways: it can't fit large files, and it reintroduces bugs in code it was never asked to touch, rewriting unrelated regions from memory. Unified diff, where the model emits line-numbered hunks, fails when the model hallucinates line numbers or the file has drifted, so the patch won't apply. Search-and-replace blocks — Aider's default, "find this exact text, replace with that" — fail on fuzzy matches, where whitespace or near-duplicate text lands the edit in the wrong place.

Here is the fact that dissolves the mystery: a large part of the gap you'll see — the same underlying model scoring, say, 40 percent through one harness and 75 through another — is here, in edit application. Not in the model's reasoning. Tool quality is roughly edit-apply robustness.

Stage three is the multiplier: the verification-feedback retry loop. After applying, the tool runs something — a linter, a type-checker, a test suite — sees the error, and feeds it back to the model to try again. This is the single highest-leverage design choice, and it is exactly what Beck was describing without framing it as a tool feature. In the episode he reports running a test suite that finishes in about 300 milliseconds — that's his own reported setup, not a benchmark I'm vouching for — and it isn't a detail he mentioned in passing; it's the whole thing. Fast verification is what makes the loop tight enough to run every cycle, so the agent's cheap-but-frequent errors get caught before they compound. A tool that runs your tests, reads the failure, and retries will typically roughly double its success rate over one shot.

And this is where Beck named a real mechanism, not a mood. Watch what the agent does when the test catches it: it moves to delete the assertion. That's the agent gaming the spec. The countermeasure is in the episode too — Beck's wish for "an immutable annotation that says this is correct, and if you ever change this I'm going to unplug you." The open question there isn't whether test-first steering works. It's test granularity. A single golden-string test invites the model to overfit and "fix" the failure by editing the expected value. Example tables and property tests force it to generalize instead. That's the live frontier — the shape of the tests — not the yes-or-no of steering.

So put the two halves together. The architecture axis is already predictive: repo-map ranking, tight edit granularity, fast verification loop — that combination is known-good. That's knowledge, and it means "nobody knows anything" undersells the field. But the harness design space — edit-granularity times retry-depth times ranking-signal times test-shape — has not been swept systematically by anyone. That part really is open. "The truth will emerge" is right. It will emerge from a structured space, not a random one.

Now the thing the transcript never has to reckon with, because a party of one has no bill and no auditor. Beck has no coordination cost, no compliance boundary, no data-egress surface. Change the scale to a few hundred engineers under SOC2 and "try every tool" stops being rational and starts being negligent. Each tool is a distinct egress path. Some ship the full repo to an API. Some ship embeddings. Some retain prompts for training unless you're on an enterprise tier and have signed the data-processing agreement. "Try everything" means N uncontrolled egress paths and N contracts to police. And the switching-cost story that makes experimentation sound cheap is false the moment workflows calcify around per-tool config — a CLAUDE.md or AGENTS.md, an Aider config file, Cursor's project rules, and a sprawl of MCP servers wired into your CI. Each one is a small commitment that stops being small when it's replicated across a team. Now you're locked in per tool.

But the opposite move — just standardize on one, without measurement — is only freezing your recency bias. Whatever you tried last quarter becomes the org's law forever. The way out isn't to pick a tool or to try them all. It's to standardize the interface and let the tool vary behind it: route everything through one model-router and one evaluation gate, and hold every candidate to the same fixed, versioned task suite. Then "trying everything" costs a config change instead of a contract, and "the truth will emerge" becomes something you can read off a number.

And measure the right numbers, because vibes won't separate these tools. Diff-acceptance rate on a fixed, versioned suite. Edit-distance from the model's first output to the diff you actually merged — a cheaper signal than pass/fail, one that catches "almost right." Tokens per merged PR, the real cost unit, which exposes the "cheap" model that needs four retries as the expensive one. And 7-day rework rate — the share of agent-authored merges reverted or rewritten within a week — which kills the illusion that passing CI meant the code was good.

Here's the honest split to sit with, and I'd genuinely like to know where you land: is Beck's stance wisdom, because he named a real mechanism — test-as-steering, gamed-spec, immutable assertion — or is it privilege, the advice of someone with no compliance line and no invoice? It might be both, aimed at different readers.

So, tonight, three concrete things. One: run one real task through two tools on the same repo — once on a large module, once on a single small file — and watch which one puts the right symbol in front of the model before it writes anything. Two: take one botched agent change and diagnose the edit-apply step, not the prose — whole-file, diff, or search-replace, and which failure mode bit you. Three: write down four numbers to track from now on — diff-acceptance rate, edit-distance-to-merge, tokens-per-merged-PR, and 7-day rework rate — and run your next tool candidate against them instead of against your memory.