This is a LensUp editorial methodology piece. It was prompted by a conversation between practitioners at Sierra and LangChain on running agents after launch; it is our synthesis of how the loop should work, not a transcript summary. Where we name specific thresholds — a 72-hour reopen window, a kappa floor, a holdout percentage — treat them as LensUp editorial recommendations and starting points to tune against your own traffic, not as settled industry consensus.
The loop is a control system with feedback latency
You are steering a metric — resolution rate, incremental conversion — by changing prompts, tool schemas, retrieval, or guardrails. The governing constraint is the lag between a change and a readable signal.
To detect a real effect, you need enough conversations per arm to reach statistical power. That takes days, not hours. This single fact explains everything downstream: why "daily" cannot mean "ship every day and read the result," and why offline replay eval exists at all — to get a fast, deterministic signal that doesn't wait on live traffic.
An easy first check: write down how many conversations per day your top intent actually gets. If it is under a few hundred, your daily read is mostly noise, and you will need offline replay to iterate at all.
Resolution rate is usually a mirror, not a measurement
The most common definition — the agent reported that it resolved the issue — is self-fulfilling. That number rises whenever the agent's language gets more confident, whether or not the user's problem got solved.
Define it behaviorally and negatively instead: resolved means closed AND not reopened or escalated within a window. We recommend a 72-hour window as a default — long enough to catch the angry return, short enough to close the loop. Tune it to your product's usage rhythm. That anchors the metric to something the agent cannot talk its way into. A user who comes back two days later was not resolved, no matter what the agent said.
An easy first check: pull 50 conversations your agent marked resolved and see how many reopened or escalated within 72 hours. That gap is your metric inflation.
Calibrate the judge before you trust the trendline
If an LLM scores resolution — LLM-as-judge — the judge itself is code, and code drifts. Edit its prompt and you have changed your ruler, not your agent.
So pin the judge to a human-labeled golden set and measure agreement with Cohen's kappa, per intent bucket. Per bucket matters. A judge can hit strong agreement on password reset and weak agreement on billing dispute, and the aggregate hides the weak spot. As a LensUp rule of thumb, hold each bucket above roughly 0.7 kappa before you let that judge drive decisions — treat that as a floor to justify, not a law.
Re-run the judge on a frozen golden set every time you touch the judge prompt. If the golden-set score moves, you changed the measurement, not the thing measured.
An easy first check: hand-label 100 conversations for one intent, run your judge on the same 100, and compute kappa. If it is below your floor, fix the judge before you trust any dashboard built on it.
Aggregate metrics lie by omission — slice per topic
Resolution rate is a mean over a mixture of very different intents. As a hypothetical: one cluster failing at 40% barely dents an 88% aggregate. You congratulate yourself on a healthy number while one workflow quietly fails half its users.
Analysis has to be sliced per topic. Which means every conversation needs an intent label. Which means you need a taxonomy that holds still — the next problem.
An easy first check: break your resolution rate out by your top 10 intents. Find the one furthest below the mean. That is your first build.
Anchor clusters to a fixed intent tree
Teams reach for embedding-based clustering to auto-discover topics. The trap is that clusters drift week to week as the topic mix shifts. You "discover" the same three insights every Monday, and your trendlines have no continuity, because last week's refund cluster is not this week's.
Fix it by anchoring the auto-labeled clusters to a fixed intent tree, and routing everything that doesn't map to an explicit unmapped bucket you review by hand. The unmapped bucket is where genuinely new failure modes live — a feature, not overflow. A stable taxonomy is the load-bearing wall that makes week-over-week comparison mean anything.
You can start by defining a fixed list of 15 to 20 intents, assigning each cluster to one, and creating the unmapped bucket. Read the unmapped pile first next week.
Rank fixes by frequency × severity × tractability
Fixing the biggest cluster is a beginner's move. The loudest cluster is often the least fixable: user typos, off-domain requests, things no prompt change touches.
Rank by frequency × severity × tractability. High volume that you cannot fix and that costs little is a bad use of a day.
And uniform sampling structurally cannot see the failures that matter most in dollars. As a hypothetical: a 0.3% fraud-refund path can account for 30% of your losses and never appear in a random sample. Oversample the tail — deliberately pull the rare, high-severity paths — or you will optimize the cheap 99% and bleed on the expensive 1%.
You can add a severity weight — dollar impact or escalation cost — to your cluster table and re-sort. Notice which cluster jumps.
Most builds are tool boundaries, not prompts
When a metric drops, the failure is often not in the natural-language reasoning. It is at a tool boundary: stale inventory from an API, a schema field the model cannot populate, a guardrail blocking a legitimate refund.
You cannot see any of this in the transcript alone. You have to join the transcript to the tool-call trace and watch two numbers: tool error rate, and unnecessary-clarification rate — the agent asking again because a tool returned nothing usable.
Worse, these regressions often come from upstream systems you don't own — a catalog update, an API version bump. That is why building never actually stops. Analysis does not "take over" so much as redirect the building; an upstream change can break you overnight regardless of how mature your loop is.
You can start with your worst intent: look at the tool-call traces, not the words. Check whether the failures cluster on one tool returning errors or empty results.
Conversion needs a counterfactual
For sales or commission agents, conversion credited to the agent is contaminated by customers who would have bought anyway. Optimize raw conversion and the agent learns to hover around people already at checkout while starving the hard mid-funnel. Attribution laundering.
The only honest target is incremental conversion. Route a slice of traffic to a holdout arm — no agent, or deflection-only — and measure the lift over that baseline. "The agent converted" and "the user would have bought regardless" are different numbers. Only one is worth money.
For sizing, we suggest carving out a 5 to 10% holdout with no agent and logging its conversion from day one. Without that baseline you cannot compute lift at all.
"Daily" cannot mean daily shipping — use a two-stage gate
A resolution-rate delta needs days of traffic per arm to reach power. By then a marketing push or a product outage has contaminated the read. Shipping every day without a gate is not iteration. It is raising the variance of your agent and calling it progress.
The real loop is two stages.
Offline replay eval on a frozen failure set. Does the new prompt fix the transcripts it targets without breaking a regression suite of previously-passing cases? This is deterministic and fast, with no live traffic needed. We recommend sizing the target set and the regression set in the low hundreds each — enough coverage to be meaningful, small enough to run in minutes.
Guarded online rollout. Stage it, with automatic rollback on a guardrail trip or a CSAT drop. Run shadow mode first: execute the proposed actions in the background without committing them, and diff against the live agent's behavior.
Proactive suggestions are hypotheses, not ship orders
A tool that surfaces improvements — call it a Ghostwriter-style assistant — is only useful if every suggestion arrives with its sample size, confidence, and provenance. Did it come from a statistically stable cluster, or from three noisy transcripts? Is that cluster mapped to your intent tree, or pulled from the unmapped bucket?
A suggestion is a hypothesis. It enters the same offline gate as everything else. If it cannot tell you how many conversations it is based on, treat it as a lead to investigate, not a change to ship.
The disagreement worth sitting with
There is one honest split among people who run these loops: what do you look at first each morning? One camp says the tool-call trace — that is where money is lost and where most regressions originate. The other says the calibrated outcome metric — tool errors that don't hurt outcomes are just noise you will chase in circles.
Both are right about the other's blind spot. The working answer: tool-trace for diagnosis, calibrated outcome metric for prioritization and gating. Which one you open first probably depends on whether your last month lost more to broken tools or to noisy priorities.
Do these three things tonight
Redefine resolution negatively — closed AND not reopened or escalated within a 72-hour window (our recommended default; tune to your product) — and pull 50 resolved conversations to measure how inflated your current number is.
Slice resolution by your top 10 intents and find the one furthest below the mean. That is your first real build, not the loudest cluster.
Freeze a replay set — the transcripts your next change targets, plus a regression set that currently passes, a few hundred each — so no fix ships without proving it fixes the target and breaks nothing.
Get those three in place and "analyze daily" stops being a slogan. It becomes a loop you can actually trust to point you at the right build — and the building, note, keeps going.
So here is the question worth answering before tomorrow morning: which number opens your morning — the tool-call trace or the calibrated outcome metric — and why?