Read the Trace, Then Build the Eval: A Data-First Playbook

The fastest way to waste three months on AI evals is to write them before you've read a single trace. Not because prospective evals are lazy — because they measure the wrong thing.

Here's the whole playbook in one line: read 30-100 real traces end to end, write down what actually went wrong, cluster those notes into failure modes, and only then turn each mode into one binary check. Metrics come last. If you're building evals for a RAG app, a support agent, or any LLM feature, do this before you buy a platform or copy a "faithfulness" score off the shelf. The rest of this explains why the order matters, and exactly what to open tonight.

Where this comes from

The core argument here is Hamel Husain's, made on the Vanishing Gradients podcast, Episode 60, "10 Things I Hate About AI Evals with Hamel Husain," published 2025-09-30 (source: https://x.com/HamelHusain/status/2067045922388005362).

His claim, stated plainly: teams write evals prospectively — up front, from imagination — without doing data analysis first. They aren't thoughtful about which evals they actually have. They waste a lot of time. And they get bitten hard by evals as a result.

That's the transcript. Everything below marked as a LensUp.ai extension — open coding, saturation curves, base-rate math, synthetic-data recipes — is our operational build-out of his data-first point, not something he said on the episode. We'll flag the seam each time we cross it.

The assumption worth killing

Everyone assumes you can design an eval the way you design a unit test: think about what should be true, write an assertion, move on. That works in software because you wrote the code, so you know its possible states.

You did not write the model. Its failures aren't designed, they're empirical. And they're long-tailed.

So when you write evals prospectively, you're sampling from your prior over failures — the things you imagine going wrong. That prior has almost no overlap with the actual failures the model produces on real inputs. You build careful checks for the failure that fires twice a month, and stay blind to the mode that dominates half your bad outputs. Your suite's coverage of real pain ends up close to random.

That's the mechanism behind Hamel's "wasted time." Not effort, not tooling. Aim.

Tonight: open your trace store, pull 20 recent outputs your users actually saw, and read them. Don't score anything. Just read.

Read the trace, not the answer

Now the second trap, and it's subtler. Even people who agree to look at the data look at the wrong slice — they read the final answer.

The final answer can't tell you where the bug is. In a RAG agent the chain is: query → retrieval → context assembly → generation. A wrong answer can be born at any stage, and a confidently wrong answer from missing context looks identical, character for character, to a genuine reasoning failure. Same symptom, opposite fix.

You only recover the cause by reading the whole trace: the retrieved chunks, their ranks, the assembled prompt, and the generation, together. That's the causal signal aggregate metrics destroy. A "0.71 faithfulness" score has already thrown away the one thing you needed — which stage broke.

For three bad outputs tonight, don't ask "is this wrong?" Ask "at which stage did it go wrong?" You'll often find it's retrieval, and you were about to blame the model.

A method for the reading: open coding, then axial coding

Everything in this section is a LensUp.ai extension of Hamel's point — a concrete way to do the data analysis he says teams skip. He argues you must read the data; here's a disciplined way to read it, borrowed from qualitative research. Open coding, then axial coding. It transfers cleanly.

Open coding: read each trace and write a freeform note about what went wrong. No categories yet. The categories are exactly what you're trying not to prejudge — predefined buckets are the prospective trap wearing a costume.

Axial coding: once you have a pile of notes, cluster them into a stable set of failure modes. "Cited the wrong policy." "Answered a question the context didn't support." "Refused a valid request." Each cluster becomes one named mode.

Notice this is the opposite of the engineering instinct, which is to define the schema first. Here the schema is the output, not the input.

Open a plain spreadsheet. One row per trace. One column: "what went wrong, in my own words." That's it. No tool, no taxonomy.

Knowing when to stop reading

People worry this reading never ends. It ends, and — this is another LensUp.ai extension, not a transcript claim — you can watch it end.

Early traces each reveal a new failure mode. Later traces just re-confirm modes you already found. Plot new-modes-discovered against traces-read and you get a saturation curve — steep, then flat. The exact point where it flattens depends on how varied your traffic is, but the shape is reliable: the discovery rate falls off, and the last stretch of reading teaches you almost nothing. When a fresh trace stops adding a mode you didn't have, that flat part is your signal: stop annotating, start automating.

That's also your discipline against scope creep. Cap the suite at 5-10 failure modes tied to real user pain. Refuse to add an eleventh until you can point at the specific traces that motivated it.

Binary checks beat 1-5 scores

The reflex is to score outputs 1 to 5. It feels richer. It's actually noisier.

An LLM judge is a noisy annotator. On a 1-5 scale the boundary between a 3 and a 4 is undefined — humans don't even agree with themselves across two sessions on the same output. So inter-rater reliability collapses, and a metric nobody agrees on can't move a decision.

A well-scoped binary check has one sharp boundary: "did the answer cite the retrieved policy? yes or no." A boundary that sharp is one a human and a judge can actually agree on, and agreement is the whole point — it's what lets the number drive a decision. You get resolution not by adding scale points but by decomposing one vague graded question into several sharp binary ones, each mapped to exactly one failure mode.

Take one failure mode from your spreadsheet tonight and write it as a yes/no question a stranger could answer from the trace alone. If you can't, the mode isn't scoped tightly enough yet.

The base-rate trap that makes a broken judge look great

One more LensUp.ai extension, and it's the one that quietly ships to production. Suppose a failure mode fires on 5% of traffic. You build a judge, measure accuracy, get 95%. Ship it.

A judge that outputs "pass" every single time also scores 95% — and catches zero failures. Accuracy is dominated by the 95% of easy passes, so it tells you almost nothing about the thing you built the judge to find.

The fix is to stop reporting blended accuracy on rare modes and report two numbers per mode: TPR (of the real failures, how many did I catch?) and TNR (of the real passes, how many did I correctly clear?). Report the confusion matrix first. If you want one summary number on top of that, a chance-corrected agreement score like Cohen's kappa beats raw accuracy — but read it with care, because kappa also collapses when a mode is very rare, purely from prevalence. The matrix is the truth; any single number is the caption.

Build the judge from the borderline examples you found during open coding — the traces where you hesitated. Those define the boundary.

The honest exception: pre-launch, you have no data

Here's where the "always read your data" line breaks, and it's worth being straight about it. A pre-launch team has no traffic. There's nothing to sample.

So the counterpoint stands: some prospective, hypothesis-driven evals are unavoidable at the start. But the principle survives — you still don't invent metrics, you generate data to read.

This next recipe is a LensUp.ai extension, not something from the episode. Enumerate 3-4 axes: persona, feature area, difficulty, and failure-inducing edge cases (ambiguous, adversarial inputs). Take the Cartesian product of the cells, generate ~50 seeded queries per cell, and run them. Now you have synthetic traces to open-code against — the same workflow, different fuel.

Two warnings. First, if your judge and your suite are both built on synthetic traces, you've built a closed loop that agrees with itself and correlates with nothing real. Second, version and deprecate synthetic evals aggressively the moment real traffic arrives. Synthetic is a dated stopgap, not a fixture. Treat it like scaffolding you tear down.

Why the generic dashboard goes up while users complain

There's an org-level reason smart teams skip all this, and it's not stupidity. Reading traces produces no Friday dashboard. A RAGAS-style suite draws a line that goes up. Leadership funds the line.

But a generic "faithfulness" scalar is averaged over heterogeneous causes. When it drops 0.02, you can't invert it — was it retrieval, prompting, a model update, or noise? There's no gradient back to a fix. So the number gets tuned up while user-reported bugs, which live in specific modes, stay flat. The metric and the product quietly decouple, and every "improvement" becomes unfalsifiable against real pain. That's the shape of a wasted quarter — the concrete version of getting "bitten hard by evals."

The alternative is unglamorous and it works: a handful of named binary checks, each traceable to a real complaint, each pointing at the stage to fix when it fails.

Do these three things tonight

• Open your trace store and read 20 full traces — query through retrieved context through generation — and for each, write one freeform note on what went wrong. No scoring, no categories.
• Cluster those notes into 5-10 named failure modes in a spreadsheet, and rewrite each as a single yes/no question answerable from the trace alone.
• Pick your rarest failure mode and check your judge with TPR and TNR separately, not accuracy — if a "pass every time" judge would score well, your accuracy number was lying.

Here's the question worth sitting with: when you finally read your own traces, which failure mode dominated — and had you written an eval for it in advance? Go find out which one, and whether you'd have guessed it.

FAQ

How many traces do I really need before writing my first eval? Enough to hit the flat part of the saturation curve — often on the order of 100 annotated, but let the data tell you. When new traces stop revealing new failure modes, you're there.

Can I just use an off-the-shelf faithfulness/relevance metric to start? Not as your foundation. They're scalars averaged over causes you can't invert, so they don't tell you what to fix. Use them, if at all, as a rough smoke test — never as the thing you optimize.

Should I buy an eval platform first? No. Buy it after you have a taxonomy and binary assertions that came from reading traces. A platform organizes evals; it can't tell you which ones matter.

Is GPT-4-as-judge trustworthy as ground truth? No judge is ground truth. Validate it against your own human labels per failure mode and check TPR/TNR before you trust it in CI.

What about 1-5 Likert scores from the judge? Skip them and decompose into binary checks — the scale points add noise, not resolution. (Covered in full above.)

Source note

The central claim — that teams write evals prospectively without data analysis, aren't thoughtful about the evals they have, waste time, and get bitten hard — comes from Hamel Husain on Vanishing Gradients, Episode 60, "10 Things I Hate About AI Evals with Hamel Husain," published 2025-09-30 (https://x.com/HamelHusain/status/2067045922388005362). The specific techniques built on top of it here — open and axial coding, the saturation curve, base-rate / TPR-TNR analysis, Cohen's kappa, and the synthetic Cartesian-product recipe — are a LensUp.ai operational extension of his data-first point, not statements from the episode.