Build Once, Analyze Daily: The Post-Launch Loop Nobody Sets Up Right

Start with the claim, because it's the useful part and it's almost right.

Before an agent ships, the whole job is building. You write the system prompt, wire the tools, connect the knowledge base. Then it goes live, and the center of gravity moves. Now the marginal cost of a change is low — you edit a prompt, you add a document — and the daily question is no longer "how do I build this," it's "what does the traffic tell me to build next." Analysis becomes the driver. The transcript that started this splits the work into three parts — analyze, build, release — and notes that most people start with build, but once you're live, your day starts with analysis. That part is true.

Here's what that framing hides.

Analysis is the daily driver because the release layer earned you the right to iterate. The reason it's safe to change a live agent every day is that a frozen offline eval set tells you the delta of a change before any user sees it. Remove that layer and "analyze-first" quietly becomes "guess-first" — anecdote-driven prompt roulette straight to production. The real causal chain runs the other direction from how it feels: reliable eval data, then trustworthy analysis, then cheap safe building, then a daily cadence. Kill the first link and everything downstream is noise-chasing wearing a lab coat. So the flywheel isn't three equal parts. Release is the flywheel. Analysis is what it lets you spend your attention on.

Now the metric, which is where most of the damage happens.

People say "we optimize resolution rate," and it sounds precise. It isn't a fact — it's a label produced by a policy you chose. Define it three ways and you get three different numbers off the same traffic, sometimes twenty points apart. Self-reported — you ask "did that solve it?" — is inflated by people who click yes just to end the conversation. No-reopen-in-72-hours conflates "resolved" with "user gave up and went to a human somewhere else." No-human-escalation counts deflection as success, which is the degenerate strategy itself. The number is the definition. If your daily analyze step is staring at one scalar you can't pin the denominator on, you're not watching the agent improve. You're watching your labeling convention.

Tonight: write down, in one sentence each, exactly which definition of "resolved" your dashboard uses and what its denominator is. If you can't, that's the first bug, and it's upstream of the model.

Then there's root cause, which splits into two physically different lanes that people constantly collapse into one.

An agent's answer is a function of two things: whether the correct information was in the retrieved context, and whether the model reasoned correctly given that context. Those are different failures with different fixes. Context insufficient is a knowledge problem — you edit the KB, resolve two conflicting source docs, add a canonical answer. Context sufficient but wrong output is a behavior problem — prompt or tool logic. Here's the trap. If you don't tag, per failed turn, "was the correct answer even in the retrieved context," you'll bucket a retrieval miss as a hallucination, patch the prompt to memorize a policy, and create something that breaks the day the policy changes. A large share of what a lazy taxonomy files under "model error" is actually two conflicting docs in the knowledge base.

So the fix is an ordering. Before you assign a failure to tool-error, hallucination, or premature-handoff, you tag context_sufficient: yes/no. That one dimension decides which lane the failure lives in. Without it, your clusters are lying to you about why things break.

And there's a wrinkle even the KB people don't get to wave away. A knowledge-base edit feels like content, not code — so people ship it without an eval gate. But a canonical policy answer that fixes cluster A can poison retrieval for cluster B, because adding or changing a document changes the candidate set and the nearest-neighbor ranking for a query — so which documents get recalled for unrelated queries shifts, even though the answer you added never mentions them. A KB edit is still a release. It doesn't skip the gate just because it looks like writing.

Now the judge, because you're probably measuring all of this with an LLM.

The judge is a model. Its labels have error, and worse, they have drift. So you never treat it as ground truth. You calibrate it against a human-labeled gold set and you measure inter-rater agreement — Cohen's kappa — not "accuracy." Accuracy lies here: if 80% of your traffic is the happy path, a judge that just says "pass" every time scores 80% accurate and is completely useless. Kappa corrects for that. As a practical operator threshold, treat something around 0.7 as workable — it's an editorial line you draw, not a settled law — and hold your own bar wherever the cost of a bad label sits. And here's the part people forget: when you upgrade the judge model, kappa can silently fall from 0.7 to 0.5, and now your entire daily flywheel is optimizing against a broken ruler and reporting green. Re-measure the judge every time you swap the judge model.

Tonight: pull fifty failed transcripts, have a human label them, and compute your judge's kappa against that set. If it's under 0.6, your metrics have been fiction, and this is the highest-leverage hour you'll spend this month.

This all gets sharper — adversarial, actually — the moment money is attached to the metric.

Under outcome pricing, where the agent earns a commission on a sale or a fee per resolution, the optimization target has a live incentive gradient. Any gameable proxy will get gamed toward the cheapest green path. Optimize resolution rate and the agent learns to resolve by surrender: deflect to the FAQ, refuse the edge case, close the ticket. The dashboard goes green while CSAT and revenue quietly bleed. Goodhart's law stops being an accident and becomes a strategy. The only structural defense is to stop optimizing a single number and optimize a contract — a primary business outcome plus a fixed set of guardrail counters with hard thresholds, so the cheapest path to green is blocked because it trips a guardrail. Resolution rate up, but escalations-after-close and refund-reversals held under a line.

Which is exactly why a proactive improvement suggestion, without a backtest, is malpractice.

A tool that says "change this prompt to lift resolution four points" is worth nothing until it replays your historical transcripts and shows the delta on the target metric and on the guardrail counters. And the thing it's changing isn't "the prompt." Your unit of iteration is a versioned bundle: system prompt, tool schemas, retrieval config, model snapshot, pinned together. Only a bundle lets you attribute a metric shift to a specific change. Change three things loose and you've learned nothing about which one moved the number. Freeze an offline eval set of replayed real conversations, run every candidate bundle against it before prod, and watch every cluster — because fixing cluster A while silently regressing cluster B is the default failure, not the rare one.

The daily job, then, isn't vibes. It's a nightly clustering run: embed the new failure transcripts, assign them to your taxonomy, and surface the cluster with the highest volume times severity — not the loudest anecdote in your inbox. Stratify your human review so the rare, expensive modes — a wrong refund policy, a payment error — get oversampled, or the 80% happy path drowns them and you never see the failure that costs you a chargeback.

So here's the question worth arguing in the comments, because it's genuinely unsettled: when a cluster is clearly a knowledge gap, is a KB edit the cheapest first fix — or is it just a release in disguise that deserves the same regression suspicion as a code change? People who run these systems don't agree. I'd like to know which way your traffic pushes you.

Three things to do tonight: ① Write down the exact definition and denominator of your "resolution" metric — one sentence. ② Label fifty failed transcripts by hand and compute your LLM judge's Cohen's kappa; if it's under 0.6, fix the ruler before anything else. ③ Add one dimension — context_sufficient: yes/no — to your failure tags, before any other bucket, so you stop routing retrieval misses into the prompt lane.