The 0.82 That Means Nothing: What Software Engineers Get Wrong About Evals

So there's this number.

You ran an off-the-shelf eval framework — pick your favorite, ragas, some LLM-judge harness — and it handed you a "faithfulness score" of 0.82. Green enough. You shipped. And here's the thing nobody in the standup asked: that 0.82 was produced by a judge that was never once checked against your labels, on your task, for your definition of correct. It's a confident number pointing at nothing.

The machine learning engineer Hamel Husain has a provocation for this. He calls it the "revenge of the data scientist" — the idea that as everyone raced to build LLM apps, the people who quietly turned out to be best at evaluating them came from statistics, not software engineering. We're going to take that provocation apart, because it's half right in a way that's more useful than either the slogan or the backlash. The real claim underneath it isn't about job titles. It's about a specific skill set — data literacy — that a lot of AI teams quietly threw out.

Let me show you the machine, because once you see how it runs, the conclusion writes itself.

Start with why you can't just write tests.

When you build normal software, you write the test against the spec before you write the feature. You know what "correct" is. LLM outputs are non-deterministic — you genuinely do not yet know how your system fails. Write your assertions up front and you'll test the failures you imagined, not the ones that actually happen. So the loop runs backwards: observe first, measure second.

The real first move is qualitative. Pull 100 to 200 real traces. Sit with them. For each one, write a plain-English note on what went wrong — that's open coding. Then cluster those notes into ten to fifteen recurring failure modes — that's axial coding. Now you have a failure taxonomy specific to your product: "drops the second file in a multi-file edit," "invents a citation," "gets the tone wrong on refunds." Every one becomes one targeted check. This is the step off-the-shelf frameworks skip — they hand you "helpfulness" and "faithfulness" before you know your own failure distribution, which is exactly why the numbers feel like theater.

Tonight: open your trace log, grab the last 50, and write one sentence per trace on what broke. That's it. You'll have half a taxonomy by the time you're done.

Then tier your checks by cost. In practice, once you've read enough traces, a large share of your real failures turn out to be catchable with deterministic, code-based assertions — no LLM in the loop. Does the patch apply. Does the JSON parse. Does it compile. Did it stay inside the schema. Did it emit a forbidden token. You reserve the expensive LLM-as-judge only for the genuinely subjective residual — reasoning quality, tone, instruction-following — where code can't decide. If you're paying a model to grade "did this parse," you're doing it wrong.

Now the crux — where the statistical mindset quietly wins the argument.

An LLM judge is not an oracle. It's a noisy binary classifier. Treat it like one. Before you trust a single number it produces, you validate it against human labels — the same way you'd validate any classifier.

Here's the trap that catches almost everyone. Say 95% of your outputs actually pass. Now build a lazy judge that just says "pass" every time. Its raw agreement with humans? 95%. Looks fantastic. Carries exactly zero information. It knows nothing; it's parroting the base rate.

That's why raw agreement is a lie, and why you need a chance-corrected number — Cohen's kappa for two raters, Krippendorff's alpha when you've got several or missing labels. Kappa subtracts out the agreement you'd get by luck given the base rate. A judge that looks like 90% raw agreement can have a kappa near zero. One I'd trust sits at 0.7-plus. Down around 0.35 to 0.4, the judge is measuring its own bias, not your model's quality. Judges get killed for this all the time — great on raw agreement, garbage on kappa.

Tonight: take one judge you're already running. Hand-label 50 of its verdicts yourself. Compute Cohen's kappa — it's four numbers in a confusion matrix, a 10-line script. If it's under 0.6, you've been shipping on a number that doesn't know anything.

And even a decent judge is biased — here's the part nobody corrects for.

Your judge has a true-positive rate and a false-positive rate that aren't perfect, which means the pass-rate it reports is systematically off from the real one. If your judge catches 90% of true failures and false-alarms on 8% of good ones, your headline "82% pass" is not the true pass rate. You can correct it — you back out the true prevalence from the observed rate using the judge's own sensitivity and specificity, a standard prevalence-estimation correction. Almost no team does this step. They report the raw judge output as if it were ground truth. It's measurement error dressed up as a KPI.

The number also needs an interval — sized to the decision you're making.

A pass-rate is a proportion estimate, and proportions on small samples are wild. Twelve examples, ten pass: your observed rate is 83%, but the 95% confidence interval is roughly plus-or-minus 20 points — a range so wide it spans "shaky" to "great." You cannot make a ship / no-ship call on that. Yet engineers do it constantly, staring at an n=12 dashboard that reads "83% good" with no error bar in sight. The fix is to compute your sample size before you label, so the interval is tight enough to decide — and to re-derive it per slice, because slicing shrinks your sample per cell.

Tonight: take your last eval run, count the actual n, and drop the observed rate into any Wilson interval calculator. If the interval is wider than the difference you were about to act on, you don't have a result yet. You have a vibe.

And here's the part that makes all of it moot if you get it wrong: the eval set is a dataset, and datasets rot.

Three ways it goes bad. Leakage — your eval items overlap the training distribution, so scores inflate for reasons that have nothing to do with quality. Peeking — you inspect your golden set every sprint and quietly tune to it, until it stops generalizing to anything but itself. Drift — production traffic shifts, and last quarter's failure modes aren't this quarter's, so your beautifully-labeled set is now measuring a product that no longer exists. Your Q1 taxonomy doesn't cover the agent behavior you shipped in Q2.

The killer, and the reason you slice everything: Simpson's paradox, live in production. Simpson's paradox is when a trend that shows up within several groups of data disappears or reverses once you combine the groups — the aggregate tells a story the subgroups flatly contradict. In an eval that plays out like this: your overall pass-rate climbs — celebration in the channel — while your highest-value enterprise cohort is regressing underneath the average. The easy majority of traffic drowns out the rare, expensive, mission-critical cases, so the combined number moves up even as the segment you care about moves down. If you don't stratify your eval set across failure modes and user segments, and slice every metric by segment, you will ship a regression to your best customer and toast the overall number that hid it.

Tonight: take one metric you report and break it out by your top three user segments. If any segment moves opposite to the average, you just found the thing your dashboard was hiding.

So here's the honest synthesis — including where the framing goes too far.

"Revenge of the data scientist" is bad politics. It makes engineers defensive, and the truth is a lot of excellent evaluators come from product and engineering — the person who wrote the feature spec often knows better than anyone what "correct" means for that task, while a data scientist parachuting in writes a generic "helpfulness" judge that misses the actual bug. You don't need a DS title to read 150 traces with discipline.

But look at the list of skills the machine actually demanded: chance-corrected agreement, confusion-matrix bias correction, proportion intervals, stratified sampling, distribution leakage, Simpson's paradox. That is not a software-engineering skill tree. That is textbook applied statistics and dataset craft. The bottleneck in eval quality sits exactly where those skills live — and on a lot of AI teams, those skills got benched.

So the fight isn't data scientists versus engineers. It's whether anyone on your team is applying the statistical mindset at all. That's the useful core of Husain's provocation, once you strip the tribal part off it.

And there's a real counter-argument here. Front-loading measurement theory can kill the loop's velocity when a bug you can see in 20 traces just needs an assertion and a shrug. So where's the line for you — day one, or only at the close A/B calls? I'd genuinely like to know where you draw it.

Three things tonight: ① Pull your last 50 traces and write one sentence each on what broke — that's your taxonomy. ② Hand-label 50 of one judge's verdicts and compute Cohen's kappa; under 0.6 and it's been guessing. ③ Break one reported metric out by your top three user segments and look for the one that moves the wrong way.

That's the point. Not the title. The habits.