Off-the-Shelf LLM Eval Metrics Are Lying to You

That faithfulness=0.87 on your dashboard is not a measurement. It's another LLM, prompted by a stranger, grading your app against a definition of "correct" you never read. And nobody put an error bar on it.

Here's the short version. Off-the-shelf eval metrics — RAGAS, DeepEval, and friends — and outsourced data labeling produce numbers that look like signal and are mostly noise plus somebody else's assumptions. The fix isn't a better metric. It's boring: read about a hundred raw traces yourself, turn what you find into a list of specific failures, build one small judge per failure, and check each judge against human labels before you trust it. Do that and the number means something. Skip it and you're optimizing a black box against a rubric that never learned what your users do.

This is for founders and PMs shipping an LLM feature who have a metric they don't quite believe. By the end you'll know why not to believe it, and what to do about it tonight.

What a canned metric actually is

Start with a dumb question. When RAGAS hands you faithfulness=0.87, what got measured?

Nothing got measured. A metric like faithfulness or answer_relevancy is one of two things: another LLM prompted to grade your output, or an embedding-similarity heuristic. It's a judgment wrapped in a name that sounds like a ruler. That naming is the trick. "Faithfulness" feels like temperature — an objective property of the answer. It isn't. It's an opinion, generated by a model, following a prompt someone else wrote.

The same thing is true one framework over. DeepEval ships answer-relevancy and other LLM-as-judge metrics with fixed rubrics baked into the prompt, and the exact calibration problem carries with them: the judge encodes a default notion of "relevant" or "faithful" that has never seen your product's constraints. Different logo, same borrowed opinion.

So you inherit three problems you can't see.

The grader is uncalibrated. The judge prompt encodes a stranger's definition of faithful. Your product's notion of a correct legal citation, or a medical answer that should hard-refuse instead of hedge, was never in that prompt. You imported a foreign rubric and started calling it ground truth.

The number is a compression. One 0–1 scalar collapses "retrieved the stale version of the contract," "invented a citation," and "correct but rude" into a single value. When that number moves, you can't tell which failure moved. And those failures have opposite fixes — one is a retrieval bug, one is a generation bug, one is a tone prompt. The compression is the information loss.

Why 0.87 is unfalsifiable

Here's the step most people stop before. An eval score is meaningless without the base rate of the thing it claims to catch.

Suppose hallucinated citations happen in 10% of your answers. A broken judge that always outputs "faithful" — one that never fires — scores about 0.90 on that corpus. Your real, working judge scores 0.87. The broken one wins. You'd ship the null judge and never know, because 0.90 looks great on a dashboard.

That's the whole problem in one example. A high score on a corpus where the failure is rare tells you nothing. You need the prevalence, and you need to know how often the judge is right when the failure is actually present versus absent.

There's a second thing missing: an error bar. Almost nobody bootstraps a confidence interval on their eval scalar. Public benchmarks are known to carry several-percent label-error rates, so any "2-point improvement" you're celebrating may sit entirely inside the noise floor of the labels themselves. A number without a CI isn't a result. It's a decision made on noise.

Why this breaks LLM apps and not normal software

You might think: we test software all the time, why is this different?

Because normal software has a deterministic spec. Input goes in, one known-correct output comes out, you assert they're equal. Done. LLM output is distributional and open-ended. "Correct" is a judgment, and now the judgment itself has to be measured — which is the thing everyone forgets.

The common mistake is writing evals prospectively: sitting down before you've looked at real outputs and inventing the assertions you think you'll need. That bakes your guesses about failure modes into the eval and misses the actual distribution of what your users do. You've built a very confident test for problems you imagined instead of the ones you have.

The correction has a name worth carrying: error analysis before metric design. Look at data first, decide what to measure second. Not the other way around.

Why you can't outsource reading the traces

Now the part that stings, because outsourcing labeling feels responsible.

Reading raw traces yourself is a discovery activity, not a QA activity. When you sort a hundred real traces by nothing in particular and read them, you find categories no rubric anticipated: users pasting the wrong document, the retriever pulling last quarter's contract, the model hedging when policy says it must hard-refuse. None of that is on anyone's spec. You learn it by looking.

This open-loop vs. closed-loop distinction tracks the ground-up practice Hamel Husain, who has written extensively on LLM evaluation, describes: you don't start from a metric, you start from the traces and let the failure taxonomy emerge. A labeling vendor is a closed loop. They can only find failures already written into your guideline. Hand them the work first and they'll faithfully confirm the failures you already knew about while the unknown ones stay invisible. Reading traces yourself is an open loop — it's product discovery. Delegate before you've done it and you've optimized a rubric that can never learn.

LensUp's read is simple: the label matters less than the move. Look at your data before you trust a number.

The pipeline that produces a number you can trust

Here's the sequence that actually works. It's not glamorous. Hamel Husain, who has written extensively on LLM evaluation, describes it as a ground-up process — open-code the traces first, then build your judges from the failure taxonomy you actually discover, rather than the one you assumed.

Open-code about 100 traces. Read them, write free-form notes on what went wrong. A hundred is enough to saturate the common failure modes. You're not doing statistics yet — you're building a codebook.

Axial-code into a failure taxonomy. Group your notes into discrete, binary failure modes. Each is present or absent: wrong-doc-retrieved, stale-retrieval, hallucinated-citation, over-hedge.

Build one binary judge per failure mode. Not one global "quality" score. One classifier per specific failure, because their fixes diverge.

Validate each judge against held-out human labels. Report TPR and TNR (or Cohen's κ) per failure mode, with the base rate attached. A judge that doesn't agree with humans doesn't get near a dashboard. The judge itself needs evals — that's the recursion people skip.

Scale with stratified sampling. Never hand-label millions. Let the validated judge grade the bulk, and have humans re-review only the band where the judge is low-confidence or disagrees with itself. That's coverage without self-deception.

Report per-slice with CIs. Break scores down by intent, by retrieval source, by document type. Bootstrap a confidence interval. Never trust a lone aggregate — aggregates hide exactly the distributional failures that matter.

LensUp's read on outsourcing: if you do it after this, do it right — expert-authored guideline, then an adjudicated gold set that includes why-notes, then contractors gated at over 90% agreement with gold, then continuous re-injection of gold to catch drift. A throughput SLA with no gold-agreement metric attached is meaningless. And note what the 90% gate buys you: consistency, not validity. Validity comes from the taxonomy the expert wrote upstream. Garbage taxonomy, garbage gold, 90% agreement on the wrong thing.

When off-the-shelf and outsourcing are actually fine

The honest counterpoint, because it's real: canned metrics and external annotators are efficient at scale. Hand-review doesn't run forever.

But the order is the whole game. Generic metrics are fine after you've established that they measure your construct and you're reporting CIs. Outsourcing is fine after your taxonomy and gold set exist. Skip the "after" and you haven't saved time — you've scaled a biased estimator and are selling the tightness of that bias as confidence. Efficient, and wrong, faster.

There's one genuinely open question even among people who do this well: how far up the query volume the founder personally stays in the traces. Some say forever, sampled. Some say hand off execution early and keep the founder only on the error-taxonomy review. The implication for product teams is that the one place you can't fully leave is the band where the judge and the human disagree. That's the irreducible human loop.

Where would you draw that line for your own product? That's the real decision, and I'd like to hear how people set it.

Do these three things tonight

• Pull 100 raw traces and read them yourself, sorted by nothing. Open your logs, dump 100 real input/output pairs into a spreadsheet, write a free-form note on each. No rubric.

• Turn those notes into 5–10 binary failure modes, then check whether your current dashboard metric even names any of them. If your metric can't tell you which of those 10 moved, it's compression, not signal.

• Take one failure mode, write a judge for it, and score that judge against 30–50 human labels. Report TPR, TNR, and the base rate. If the judge can't beat a null judge that always says "fine," you've found your real problem.

FAQ

Are off-the-shelf LLM eval metrics reliable?

Not out of the box. They're LLM-as-judge or similarity heuristics carrying a stranger's rubric, no base rate, and no error bar. They become reliable only after you validate them against your own human labels and report per-slice with confidence intervals.

How do I validate LLM-as-judge calibration?

Validate the judge against a held-out set of human labels it never saw. Report true-positive and true-negative rates for each failure mode separately, with the base rate attached, and compare against a null judge that always predicts the majority class at the corpus base rate. If your judge can't beat that null baseline, its high score is coming from prevalence, not from actually detecting the failure.

Should I outsource AI data labeling?

Yes — but not first. Read traces and build your failure taxonomy yourself, create an adjudicated gold set, then gate contractors at over 90% gold agreement with continuous re-injection. Outsource execution, never discovery.

How many traces do I need to read?

About 100 for the first taxonomy pass. That's enough to saturate the common failure modes. You're building a codebook, not sampling for statistics — the statistics come later, once the judges are validated.

What single number should I trust?

None, alone. Trust per-failure-mode judges, validated against humans, reported by slice, each with a bootstrapped CI. If a delta is smaller than your labels' error rate, treat it as unmeasured.