Here's the short version. A unit test asserts that f(x) equals one correct value. An LLM at temperature above zero doesn't produce one correct value. It samples from a distribution of acceptable answers that differ by wording, ordering, and formatting. So assertEqual on generated text fails even when the model was right.
Evals are a measurement discipline, not an assertion discipline. You calibrate instruments and estimate rates. The one exception is the deterministic plumbing layer, where real contracts still exist, and where you should keep every hard check you have.
That's the whole idea. The rest is why it's true and what to do about it.
Where this comes from
Two of the sharpest framings below come from Hamel Husain, who has argued at length that reading your own traces beats picking a framework, and that an LLM judge is a sensor you have to validate against humans. Those specific points are his; I quote them inline where they land. The rest is standard ML evaluation practice — pass rates, confidence intervals, inter-annotator agreement — stitched together into one workflow. Take it as a synthesis, not a transcript.
You're comparing a sample to a point
A unit test works because the output space has one true point. assertEqual(add(2,2), 4) is valid because there is exactly one right answer. That's a contract.
Generated prose has no single point. Ask a model to summarize a ticket and there are hundreds of acceptable summaries: different sentence order, different hedging, a comma here. The correct answer isn't a point, it's a region. When you run assertEqual against generated text, you're comparing one sample against one point out of a region. It fails frequently, and not because the model was wrong.
This is how the software instinct quietly poisons the suite. Your tests go red for the wrong reason. So you do one of two things: delete the tests and lose coverage, or loosen them into "contains the word 'refund'" until they mean nothing. Both are failures, and both come from forcing a contract onto something that doesn't have one.
The fix is a rate, not a boolean
If the target is a distribution, the right object is a rate, not a pass/fail. You draw n samples, grade them, and estimate P(acceptable) with a confidence interval.
Concretely: run the same input 20 times, grade each, and report the fraction that passed. Say 17 of 20 pass. That's 85% as a point estimate — but with only 20 samples the interval around it is wide. Run it through an exact binomial method like Clopper-Pearson and 17/20 gives you roughly 62% to 97% at 95% confidence. The lesson isn't the exact bound; it's that a single green checkmark hides all of this. A small sample gives you a shaky estimate, and you need more runs before you trust a difference between two versions.
Try this: take one input your app handles, run it 20 times, and eyeball how much the outputs vary. That spread is your real signal. Hold that thought. It comes back at the end.
The judge is a sensor you forgot to calibrate
Some outputs aren't machine-checkable. Was the summary faithful? Was the tone right? You can't regex that, so people hand grading to another LLM, an "LLM-as-judge." Hamel's framing is the useful one here: treat the judge as a measuring instrument, not an oracle.
The trap is that people treat "GPT-4 said 8/10" as ground truth. That's just the un-validated assertion moved up one level. You wouldn't trust a thermometer you never checked against a known temperature.
To trust the judge, characterize its error. Start with a small human-labeled set — outputs a person marked good or bad — and expand it as your criteria stabilize. Then measure the judge's true-positive and false-positive rate against those labels. A judge with a 30% false-positive rate will happily wave garbage through, and you won't know until prod tells you.
Human agreement is the ceiling
Here's the part almost everyone skips, and it sits upstream of the judge.
Before you check the judge against humans, check the humans against each other. Have two people label the same 50 outputs and compute their agreement. Cohen's kappa is the standard number. If two humans can't agree on whether a response is good, your criterion is under-specified. The judge is then fitting noise, and no amount of prompt tuning fixes it.
Human agreement is the ceiling on judge reliability. If your labelers disagree, you don't have a judge problem, you have a rubric problem. Rewrite the rubric until humans agree, then build the judge.
The step nobody wants to do first
Many failed eval efforts die before any metric exists. A common tell is when step one is "pick a framework," Ragas, DeepEval, promptfoo. You can't write a rubric for a failure you've never seen. This is Hamel's point, put bluntly: the first move is not tooling, it's reading.
The unglamorous move that actually works: open 30 to 50 real traces from your app and read them by hand. Write free-text notes on what went wrong. Cluster the notes. Codify the top two or three into evals. That cluster list is your failure taxonomy, and it does double duty. It's the judge's rubric and the labeling schema for your gold set.
Picking a framework first means you build precise measurements of failures you don't have while missing the ones you do. The ordering is what matters: error analysis, then taxonomy, then judge. People often do it backwards.
Try this: open 20 real traces and write one plain sentence per trace about what's off. Don't categorize yet. Just read.
Don't throw out the plumbing
Now the overcorrection, because half of you are about to rip out schema validation, and you shouldn't.
The stochastic argument applies to the natural-language layer. The plumbing is deterministic, and it's a genuine contract:
assert JSON parses against schema— a malformed tool call takes down prod, full stop.assert tool_called_with(valid_args)— argument validation against the function signature.assert citation_id exists in corpus— a cited source that doesn't exist is a hard bug, not a judgment call.assert retrieval returned exactly k docs— a plumbing failure, not a quality one.
These are two-line checks that belong in fast CI as hard gates. They have one right answer, so they are contracts. Keep them.
Rule of thumb: if the answer is a single correct value, assert it deterministically. If the answer is a region, measure a rate. Two tiers, not one.
Frozen fixtures vs living samples
A software fixture is frozen so the test is reproducible. But your prod input distribution drifts as usage changes, so a frozen eval set slowly measures a world that no longer exists. Your green score decouples from real quality.
The instinct to freeze and the need to refresh collide. The resolution is two datasets, reported separately:
- A regression set — frozen, versioned, reproducible. Catches known bugs coming back.
- A fresh sample — rolling, stratified by input distribution, pulled from recent prod. Catches drift and new failure modes.
Report them together and you've made a mess, because your week-over-week score isn't comparable when the underlying set changed. Keep them apart. Regression score is "did old bugs return." Fresh score is "how are we doing on what's happening now."
The cleanest flip: variance is the measurement
The single software instinct that sabotages people most: a flaky test is a bug to eliminate. You retry until green.
In evals, do that and you've deleted your own signal. Variance isn't noise here. It is the reliability of your system. If you run at temperature above zero and retry until green, you've measured your luckiest run, not your system.
So run n samples, track a pass rate with a confidence interval, and break it down per failure mode. A single aggregate number with no CI and no per-mode breakdown is worse than useless, because it looks like knowledge.
That spread you eyeballed earlier? That was the whole point. In a stochastic system, the wobble is the answer.
A practical starting point
Three concrete moves, in order:
- Read 20 real traces and write one plain sentence each about what went wrong. No framework, no metric, just notes. This is your future taxonomy.
- Run one input 20 times and record the pass rate, not pass/fail. Look at the spread on purpose, and remember that 20 runs gives you a rough estimate, not a precise one.
- Audit your CI: move every schema, tool-arg, and citation-existence check into a fast deterministic gate, and pull every "does this text look right" assertion out of that gate. Those two things do not belong in the same tier.
One honest question back to you: when your eval suite last went red, did you fix the model, or did you quietly loosen the assertion? That answer tells you which tier you were really in.
FAQ
So are unit tests useless for LLM apps?
No. They're correct for the deterministic layer: schema, tool calls, retrieval structure. They break only when you point them at generated natural language, where there's no single right answer.
Why not just use a stronger judge model and skip calibration?
A stronger model can still have a high false-positive rate on your specific task, and you have no way to know without a human-labeled set. Calibrate first. A strong judge you haven't measured is still an uncalibrated sensor.
What's the fastest signal I have a paradigm problem, not a prompt problem?
You keep loosening assertions until they pass, or you retry until green. Both mean you're forcing a contract onto a distribution. Switch to rates and confidence intervals, and the problem usually reframes itself.
Where should error analysis notes live?
Anywhere you can cluster them. A spreadsheet is fine. The tool doesn't matter. Reading real traces before writing any rubric is the part that matters.