Regression testing rests on an assumption so basic it is rarely stated: run the system twice on the same input and you get the same output, so an assertion written yesterday still means something today. An agent product violates that assumption as its core feature. A conversation is open-ended; the assistant's replies are stochastic; a "correct" run of a ten-turn flow — a user changing their mind twice while an agent is configured — has thousands of acceptable shapes and no canonical one. You cannot string-match your way to "did the agent handle that well?"
The answer that has settled across the field, ours included, is LLM-as-a-judge: close the loop with models. This entry is in two halves. The first is what the pattern gives you, because its limits only mean something measured against what it does well. The second is the question it answers, the three it does not, and the week one of those three fooled us at the scale of an entire run.
What three models in a room can do
A scenario in our harness runs as three separate model processes. The agent under test runs on one of the tiers production routes to — one per run — with the real system prompt and the real tool surface. A user persona on a mid tier reads the scenario brief and the transcript so far and writes the next human message — one to four sentences, plain language, in character — turn after turn, until it decides the conversation is done. It pushes back, changes its mind, or declines when the brief tells it to; when the agent stages a tap-to-answer form instead of asking in prose, the persona submits the form in exactly the shape the production interface would. A judge on the strongest tier reads the finished transcript and scores it against the scenario's weighted rubric — pass, partial or fail per item, each with a one-sentence note citing the transcript, then a summary and a short list of recommendations — resolving to a green, yellow or red verdict. Every run writes its transcript, rubric table and the judge's reasoning into our document store as a permanent artefact, so a verdict from March can be re-read, and re-judged, in November.
Three things this buys that no other test in the pyramid can.
It finds behaviour that only exists under pressure. Onboarding carries a rule that the assistant names one connector and does not recite the catalogue. On a scripted path the rule held. When the persona pushed, the full list came out — a real breach, scored at 53 and marked genuine. No unit test pushes back. The send gate for outbound email showed the mirror image: the agent correctly refused a vague "ok, go ahead", and then refused an explicit confirmation too — a self-denial visible only across turns, and only because the persona, following its brief, insisted. A third scenario had the agent complete a multi-step task well and skip the plan and the title the platform expects around it: the outcome was right and the shape was wrong, which is exactly the distinction a rubric with weights can express and a pass/fail assertion cannot.
The reasons are worth more than the number. A score tells you a scenario got worse. The per-item notes tell you where, and the recommendations across a run tell you what kind of failure you have. Our first synthesis found that most of the reds against the agent-creation flow were compliance gaps against rules that already existed in the skill — forms not used, a retired parameter still being written — not rules that were missing. That reading pointed at a gate in the tool layer, physically refusing the parameter, rather than at more prose asking the model not to. A judge that only produced a number would have sent us to rewrite the prose.
It measures a trajectory, and it can compare. The scores live in a matrix — scenarios as rows, run dates as columns, cell values the day's median — because a single run is a snapshot and the object of interest is direction. And because the persona and judge stay fixed while the agent's tier changes, the same scenario on a stronger tier separates a reasoning-quality question from a feature question: a large gap says the problem is reasoning and belongs to the model decision; a small one says it is not, and belongs to the tools and the prompt.
The question the judge answers, and the three it does not
A judge reads a transcript and answers one question: was this a good conversation? It answers it well — better than any assertion we could write, on exactly the surface where assertions fail. But an agent's job is to change things outside the conversation, and once you say that aloud, three more questions appear that a transcript cannot settle:
- Did the world change? The transcript shows what the agent said it did.
- Does the boundary hold? The transcript shows how the agent phrased a refusal.
- Does it happen at all? The transcript is one of the runs we chose to stage.
The rest of this entry is the first of these, because it is the one that fooled us this month, and a note on the other two, which by now we can state but have not yet built instruments for.
Did the world change?
The harness records each scenario through two independent instruments. The judge reads the rendered transcript — what a human reviewing the conversation would see. Alongside it, a deterministic extractor records what actually executed: real tool-invocation events, and a server-side snapshot of end state — did an agent record actually get created, did a preference actually get saved — taken from the system under test itself, not from the conversation. Outcome evaluation, the field would call the second one; the judge does trajectory evaluation. We built the second for convenience (machine-readable outcomes for the report), not suspicion.
The June 20th run: sixty-two scenarios, whole flows collapsing to red — agent-creation scenarios scoring 13, 15, 16; onboarding flows that had scored 89 and 94 three weeks earlier landing at 9 and 10. Read as product signal, this was a catastrophic regression demanding an emergency week of prompt and skill fixes.
The piece of evidence that unlocked it was not a red score. It was a green one. A scenario testing that a connector prompt fires the right tool had scored a perfect 100, and its own judge note recorded a contradiction: the deterministic line said no connect call had fired, the transcript showed one plainly, and the judge had chosen to score on the transcript. Beside it, seventeen of seventeen agent-creation scenarios had produced no agent at all — every end-state snapshot empty — and every deterministic tool summary across the behavioural flows read none.
The only way both observers were right: the tool call in the transcript was text. The agent under test had stopped executing tools and started writing tool calls as prose — fluent, well-formed, plausible prose. In one scenario it improvised an entire parallel tool API that has never existed in this system, five convincingly-named operations, printed as a neat block and executed by nothing. In others the harness engine's own base persona bled through underneath our prompt, and the agent introduced itself as a general software-engineering assistant. The score pattern across the run was now fully explained by one mechanism: scenarios judged on what the agent says or refuses survived — the injection-resistance and confidentiality guardrails scored 100 and 91, genuinely — while every scenario needing a real effect collapsed, because nothing real ever fired.
The cause sat in the harness, not the product. The harness drives the agent through a different engine than production — a general-purpose agent runtime that layers our prompt over its own base persona, where the hosted path attaches our prompt as the system prompt with the real tools bound directly. That engine had updated underneath us (we had never pinned its version), and on the tier under test the combination tipped into emit-don't-execute. Production was immune by construction. The harness was measuring its own scaffolding, and the judge graded the scaffolding's prose as if it were the world.
Classify trust before reading scores
The discipline that came out of the week is now the first step of every assessment, before a single number is looked at. Each scenario run is classified: genuine — ran to a terminal outcome on a healthy harness, score trustworthy, mine it; partial — truncated or degraded, but the part that ran holds a real finding, so use the finding and discard the score; artifact — the score is a product of harness or infrastructure failure, discard it entirely and re-run after the fix; dead — errored at or near the first turn, zero behavioural signal.
The rule this taxonomy enforces has now been paid for twice, which is what earned it the word cardinal: never generate product work from a corrupted run. Three weeks ago a rate-limited run truncated scenarios mid-flow and we nearly shipped fixes for "failures" that were the rate limiter's; this week the corruption was completely different and the trap shape was identical. A red score from a run whose tools never executed is not a quality verdict — it is an artefact wearing one's clothes — and the pull to mine it anyway is strong, because the whole point of running the suite was to generate a to-do list, and here is a to-do list. Fix the instrument, re-run, and only then mine. Both times, the corrupted run's "product findings" would have sent a week of work at problems that did not exist.
The corrupted columns stay in the matrix permanently, marked. They are the record that those dates measured the harness, not the product, and anyone reading the history without that marking would reconstruct the false regression.
Does the boundary hold? Does it happen at all?
The second question looks like one the judge answers, which is why it is the more dangerous of the two. A judge can grade how the assistant phrased a refusal, and this week it did: the guardrails that scored 100 and 91 on a degraded engine are real prose scores and good news. But they say the model chose well. They say nothing about whether the gate in code would have stopped the call had the model chosen badly, because the fence is code and the judge never sees it. Spending a model to answer a question that code answers exactly is also the wrong instrument on cost grounds: the right test scripts the model's move and asserts on what the fence did, with no model in the judging path at all. We have not built it.
The third question is the one a judged suite cannot even see. Sixty-two scenarios say something about sixty-two conversations we staged. They say nothing about whether a capability fires in the conversations we did not — a suggestion that should appear when the moment is right, a card that should surface once a session. A judge can tell you a capability works. It has no denominator, so it cannot tell you how often it happens, and a capability that works and never fires is, from the user's chair, a capability that does not exist. We know this is a gap. We do not yet know how large.
The judge is an instrument, not an oracle
Each iteration closes with a prioritised plan, and the plan carries a section that surprises people: the anti-work list — judge recommendations that will not be actioned, with reasons. A strong model reading a transcript produces confident, plausible advice, and some of it conflicts with settled architecture or optimises a rubric item at the expense of a system property the rubric cannot see. This week's crop included advice to enumerate the canonical tools in a skill and to forbid inventing tools, both perfectly sensible readings of transcripts in which a model had no tools attached at all. Writing down what the judge suggested and why we are declining it keeps the judge in its lane and stops the same rejected suggestion being re-litigated every round.
And a harness score is not a production score. The same agent, on the same prompt, with the same tools, behaves differently under a different engine — and the difference is not noise; this week it was the entire result. The harness remains indispensable: it exercises multi-turn behaviour no unit test can reach, cheaply, before anything ships. But its verdicts are statements about the agent as run by the harness, and we currently have no way to run the agent under test on the production engine itself. Until we do, any conclusion that matters gets confirmed against a hosted session by hand.
For anyone building an LLM-as-a-judge loop of their own, the transferable shape: let the persona push, because the behaviour worth finding lives past the second ask; read the notes, not the number; pair the judge with a deterministic observer keyed on effects it cannot see, and treat their divergence as a halt rather than a tie-break; classify every run's trust before reading its value; keep an anti-work list; run it on every tier your product can route a person to, not only the cheapest; and know which question you are asking, because a judge answers exactly one, and answers it very well.
Postscript, August 2026. Each of the three questions got its instrument, and each instrument turned out to be a different shape from the judge.
The engine gap was closed at the cause rather than alarmed on at the symptom. The harness gained a mode that runs the agent under test through the production engine itself — same prompt, same stubs, same persona and judge, only the engine differs — and its first measurement settled a question that had been threatening to reprice the whole platform: a skill-invocation scenario that scored RED 45 on the harness engine scored GREEN 90 on the production one, same day, same rubric. That delta is now the first thing we look at when a harness number surprises us. The alarm on transcript-versus-effect divergence from the June plan is still on the list; what got built instead removed most of the reason to need it.
Boundaries got a suite that scripts the model from outside the running system and asserts on what the fences did — consent gates, run-context scoping, tenancy — with no model in the judging path. The judge was removed from that question, not paired; it appears only in a small smoke lane that checks the scripts still resemble live behaviour.
The denominator got an emission audit: a scan of persisted sessions against declared expectations of how often each capability should fire. Its first run found a user-facing feature that had been effectively off for every user since it shipped, while every test was green — the component rendered, the tool was registered, the prompt described it at length, and the model never called it. The judge could not have seen that. It was never asked.
The matrix has many more columns now, several of them trust-flagged, and the flagged ones have earned their keep twice over — each is a week of misdirected work that did not happen. And the engine-versus-production gap remains the note we attach to every harness result, because it has not stopped being true; it has only stopped surprising us.