Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation
Zhou Yu · presentation · 2026-09-07 source ↗ Reader 🖥️
Enterprise AI Adoption & ReadinessEvalsFeedback LoopsGuardrailsHuman-in-the-LoopMulti-Agent OrchestrationSynthetic User Simulation
Because static, single-turn benchmarks with fixed golden answers cannot evaluate multi-turn, tool-using agents whose correctness depends on live backend state and conversational history, teams must replace ad hoc manual testing with AI-simulated user personas that generate realistic, coverage-optimized conversation trajectories against which scenario-grounded, automated evaluation can run inside CI/CD — turning agent testing from the manual bottleneck that keeps roughly 95% of agents stuck in demo into a scalable, continuously-refreshed, self-improving production pipeline.
practical · mixed — A conference presentation arguing for a specific methodology (simulation-driven, CI/CD-integrated agent testing). It mixes expository sections (architecture of the scenario model, evaluation metrics), narrative demos (Sparky, the credit-card voice agent, the ArkSim CI walkthrough, the shopping-agent error trace), an explicit argumentative case for why the old evaluation paradigm fails, and a Q&A that surfaces unresolved edge cases.
Structure
| Part | Role |
|---|---|
| Opening framing: ~95% of AI agents never leave the demo phase despite being impressive to watch. | motivation / problem statement |
| Two demos of conversational/voice agents already in production or near-production (Walmart's Sparky, a fictionalized credit-card voice agent) — the second explicitly flagged as 'not actually in production' because of compliance constraints in finance. | evidence establishing the shape of the problem: capability exists, but compliance/evaluation is the actual blocker |
| 'Gaps in Current AI Agent Evaluation': diagnosis of why traditional single-turn benchmarks and manual friend/coworker testing both fail for multi-turn, tool-using agents. | diagnostic argument |
| 'Solution: Synthetic Users' and the ArkSim CI/CD demo walkthrough: AI agents simulate diverse end users to generate trajectories automatically, wired into a build pipeline with pass/fail thresholds. | core proposal |
| High-level architecture: a scenario is the permutation of user profile, user goal/intent (seeded by the agent's own tool capabilities), and context information. | conceptual model of the solution |
| Evaluation is split into two separate axes: (a) quality of the simulator itself (coverage/entropy, realism, cost) and (b) quality of the product agent under test (task completion, efficiency, turn-by-turn LLM-as-judge metrics grounded in the known scenario). | elaboration / defines success criteria for the whole system |
| Worked example (order-status agent) showing simulation surfacing a specific, diagnosable root cause (broken shipping-info tool, missing RAG data). | narrative evidence |
| 'Ever-Evolving Golden Scenario Set': production logs are diffed against existing simulated distributions to keep the scenario set from going stale. | extension addressing long-term maintenance / drift |
| Pearson enterprise case study and 'Self-Learning Agents' section, including a return-vs-cancel-order flow-repair example and mortgage-underwriting synthetic-document generation. | evidence of ROI and generalization of simulation beyond testing into training and continual improvement |
| Q&A: persona realism at the extremes (toddlers, senile users), soft/unverifiable goals (product recommendations), and cold-start vs. warm-start persona construction. | clarifies scope and exposes unresolved edges |
Key terms
- Trajectory entropy (tool call distribution entropy / tool call transition entropy / trajectory distance) Synthetic User Simulation
A family of quantitative diversity metrics for the *simulator*, not the product agent: how spread out the set of tools a synthetic user invokes is (distribution entropy), how varied the sequence in which it moves between tools is (transition entropy), and how far apart different simulated conversations are from each other (trajectory distance) — together used to detect and penalize a simulator that keeps generating near-identical, low-coverage test conversations. - Synthetic user / simulated user
An autonomous AI agent instantiated with a persona (demographic/behavioral profile, a goal seeded from the target agent's own declared tool capabilities, and contextual knowledge) that converses with the product agent end-to-end, over an API, to generate a realistic multi-turn trajectory — functioning simultaneously as a tester, an error-discovery mechanism, and (in privacy-restricted domains) a training-data generator. - Golden scenario set (as 'ever-evolving') Synthetic User Simulation
A curated, versioned collection of (profile, goal, context) test scenarios treated as a living CI/CD regression suite that is deliberately and continuously revised — new scenarios are synthesized whenever production logs reveal a gap between real and simulated user distributions. - Realism (as a simulator-quality axis)
The requirement that a synthetic user behave like a plausible real user pursuing a genuine goal — not merely an adversarial agent trying to trigger errors — verified via reward functions for adherence to the assigned profile/goal and distributional similarity to collected real-user data. - Cold start / warm start (applied to user simulation)
Cold start: no production interaction data exists yet, so scenarios must be bootstrapped purely from domain-expert interviews. Warm start: production data from real human-handled interactions already exists and the simulation should be built to match that distribution.
Propositions
- explicitRoughly 95% of AI agents remain stuck in the demo phase and fail to create real, deployed production value.conf: high
- explicitStatic single-turn benchmarks with a fixed 'golden' reference output become obsolete once agents are multi-turn, because a user's next question depends on the agent's prior answer, so no ground truth can be fixed in advance.conf: highsupportsKaggle is making AI benchmark creation effortless · p1 — Both independently argue that traditional/static benchmarks are no longer adequate once agents are multi-turn, tool-using, and reasoning rather than answering single fixed questions.
- explicitBecause agent tool calls touch live, time-varying backend state (account balances, order/shipment status, inventory), correctness cannot be judged from the agent's stated output alone — it must be checked against the actual state of the underlying systems.conf: high
- explicitManually recruiting coworkers and friends to chat-test an agent produces poor test coverage — non-real users, especially technical colleagues, don't represent realistic user-intent distributions — and the entire manual test cycle has to be redone from scratch every time the agent changes, because the previously collected data no longer matches the new agent.conf: highsupportsThe Production Gap: 5 Patterns for Building Long-Running AI Agents · p2 — Zhou Yu's account of why manual chat-testing fails to catch real production behavior is a concrete instance of the general 'production gap' between what demos show and what real workflows require.
- explicitAI agents can be used to simulate diverse end users — as other AI agents with distinct personas, goals, and context — that interact automatically and at scale with a product agent to generate realistic multi-turn conversation trajectories for testing.conf: high
- explicitA test scenario is fully specified by three independently variable components — user profile, user goal/intent (seeded by the target agent's own declared tool capabilities), and context information — and permuting these three yields broad, controllable scenario coverage.conf: highsupportsPodcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook · p2 — Both converge on the claim that the highest-value evaluation scenarios are built from an organization's own domain knowledge and tool capabilities, not generic off-the-shelf tasks.
- explicitSimulated-user testing can be wired directly into a CI/CD pipeline so every code push triggers building the new agent, running a fixed battery of simulated conversations against it, scoring the results on multiple dimensions, and passing or failing the build against a threshold.conf: highsupportsThe Anatomy of an Agent Harness · p7 — Both name automated, CI-integrated verification (rules-based checks plus LLM-as-judge) as what actually separates toy demos from production-ready agents.
- explicitdevelopedGrounding an LLM-as-judge evaluation in a specific simulation scenario with a known, definable goal allows task completion to be verified automatically by rule-based checks against real backend state, which is more trustworthy than asking an LLM to judge an open-ended conversation without a defined ground truth.conf: highextendsDownstream #1: Where Do Our Tokens Go? · p7 — Where the other document argues organizations need their own evals built as shadow comparisons rather than relying on public benchmarks, this talk supplies a concrete mechanism for building those organization-specific evals: simulation-grounded, verifiable scenarios.extends ←A pragmatic guide to LLM evals for devs · p5 — The other document grounds LLM-as-judge validity in a specific simulation scenario with a known goal; this article extends that with the concrete workflow for producing the human-labeled golden dataset (PASS/FAIL + critique) a judge should be built and calibrated from.
- explicitA good user simulator must be optimized on axes distinct from the product agent's own optimization target: coverage (tool-call distribution/transition entropy and trajectory distance/diversity), realism (behaving like plausible real users rather than pure adversaries), and cost-efficiency (minimum scenarios for maximum coverage) — with the number of real agent failures the simulation surfaces as the ultimate downstream success metric.conf: high
- explicitBecause real user behavior distributions drift over time (new products, policies, seasons), a static simulated-scenario set decays in relevance, so scenarios must be continuously refreshed by diffing production logs against the existing simulated distribution and synthesizing new scenarios to cover the gap.conf: high
- explicitCompliance and regulatory review — not raw model capability — is the primary bottleneck keeping consumer-facing agentic systems out of production in finance and healthcare, relative to lower-risk domains like e-commerce where similar agents already ship.conf: high
- implicitSynthetic user simulation is valuable not only for testing but as a training-data-generation mechanism for domains where real data either doesn't exist yet (a brand-new agent, cold start) or is legally unusable (PII-restricted domains like mortgage underwriting).conf: high
- implicitThe same simulation infrastructure used to find agent errors can be looped back to improve the agent — via flow/prompt redesign suggested by discovered corner cases, or via fine-tuning smaller models on synthetic conversation data — implying that testing and self-improvement are the same underlying loop rather than separate phases.conf: high
- implicitTreating agent quality as a single overall benchmark score is inadequate for production decisions; quality must be decomposed into distinct, separately-judged axes (helpfulness, coherence, verbosity, relevance, faithfulness, brand-voice adherence, task/goal completion, efficiency, call-to-action success), each potentially requiring its own calibrated judge.conf: medium
Arguments
Agent evaluation must shift from static single-turn benchmarks and ad hoc manual testing to simulation-driven, scenario-grounded multi-turn testing. strength: medium
Explicit premises:
- Static single-turn benchmarks require a fixed golden reference output, which cannot be defined once conversations are multi-turn and context-dependent.
- Live tool-call state (balances, order status, inventory) changes continuously, so no static reference output can exist to compare against.
- Manual human-tester recruitment doesn't scale, isn't repeatable across agent versions, and captures unrealistic (non-representative) user-intent distributions.
Implicit premises:
- A viable evaluation method must be repeatable across successive agent versions without re-collecting fresh human-generated test data each time.
- A test method needs to approximate the real user-intent distribution to have any external validity — otherwise passing the test says nothing about production readiness.
- It is possible, in practice, to construct AI-simulated personas that adequately approximate real user goal/intent distributions (this crux assumption is asserted via the Pearson case study, not independently validated).
Inferences:
- If simulated users can be made sufficiently realistic and diverse, manual human testing could be reduced to a final validation step rather than remaining the primary testing method — a conclusion the talk gestures at ('you only needed a more important manual testing at the end of the day') but does not fully argue for.
LLM-as-judge evaluation becomes trustworthy for agent testing only when grounded in a specific, verifiable simulation scenario rather than asked to judge an open-ended conversation in the abstract. strength: medium
Explicit premises:
- An LLM-as-judge cannot be assumed to independently know a business's own policies, documents, or ground truth.
- When the scenario is generator by simulation, the tester already knows the user's goal, so task completion can be verified automatically via rule-based checks against real backend state.
- Softer, turn-by-turn quality metrics (helpfulness, brand voice) still require an LLM-as-judge, but its reliability can be improved by seeding it with curated hard-negative and hard-positive examples drawn from generated or expert-reviewed conversations.
Implicit premises:
- Rule-based verification of a known goal against backend state is assumed to be a strictly more trustworthy ground truth than any LLM judgment, without argument.
- Curating good hard-negative/positive seed examples for judge calibration is itself treated as a tractable, non-circular task; the talk doesn't address who verifies the curator's own judgment, or the judge's judgment of the curated examples.
Inferences:
- If the judge's seed examples are themselves produced or selected using the same simulation/LLM pipeline being evaluated, the judge and the system under test could share correlated blind spots — an inference the talk's own description of the pipeline invites but never draws or addresses.
Narratives
- Walmart's 'Sparky' and Amazon's 'Rufus' shopping agents shown answering product questions, pulling in review cards, and maintaining contextual understanding across a shopping session.
→ Establishes that conversational/chat-based commerce agents are already an established, deployed production category — grounding the definition of 'agent' before the talk turns to why evaluating them is hard. - A two-call voice-agent demo: a customer applies for and is approved for a credit card, then calls back later with follow-up questions — explicitly flagged by the speaker as 'not actually in production.'
→ Implicitly supports the claim that a technically impressive, working demo is insufficient for production in regulated domains because of compliance requirements (explicit authorization, PII, actioned database changes) — the demo's very polish is the point being undercut. - A walkthrough of the open-source ArkSim tool testing a GED-exam Q&A agent: defining agent capabilities and knowledge, generating scenarios, running them in a CI pipeline on every push, and producing a five-dimension scored error report.
→ Supports the technical feasibility and CI/CD-integration of the proposed simulation solution with a concrete, reproducible workflow rather than an abstract description. - A simulated shopping-agent conversation about order 1300's shipping status, where the agent progressively fails to answer, deflects, and cannot provide a customer-service email, with the failure traced to a broken shipping-info tool and gaps in a RAG/vector-database pipeline.
→ Supports the claim that simulation surfaces concrete, diagnosable root causes rather than a bare pass/fail score, and that it can find these errors before real users encounter them. - For mortgage underwriting, synthetic paystubs, bank statements, and T4 forms are generated — including deliberately corrupted variants (skewed scans, missing pages, poor lighting) — to create corner cases without using real, PII-restricted borrower data.
→ Supports the implicit proposition that simulation is a training-data solution for privacy-restricted domains, not only a testing tool. - A rigid 'return order' flow fails because a not-yet-shipped item can only be cancelled, not returned; the fix generalizes the flow to fall back from 'return' to 'cancel' when the order hasn't shipped.
→ Supports the claim that simulation-discovered corner cases translate directly into concrete flow/prompt redesigns, closing the loop between testing and self-improvement. - Pearson, serving students across 200+ countries with differing languages, testing policies, and purchasing policies, uses Arklex's simulation over a two-year relationship to make releases 'much more rigorous.'
→ Supports the claim of real, sustained enterprise ROI at genuine scale/complexity, beyond a one-off demo.
Questions addressed
- ✅ Why do most AI agents never leave the demo phase and reach production?
Compliance review demands rigorous evaluation, and traditional single-turn, static benchmarks cannot evaluate multi-turn, tool-using, live-state-dependent agents — so agents fail compliance review, especially in regulated industries like finance. - ✅ How should multi-turn, tool-using agents be evaluated when no static ground-truth reference can be defined in advance?
By simulating a known user goal within a specific scenario, so task completion can be verified automatically against real backend state rather than compared to a fixed reference answer. - ✅ How can realistic agent testing be scaled without an army of manual human testers?
By instantiating synthetic user agents as an on-demand API endpoint that can be spun up in parallel, at any time, with configurable personas. - ✅ How do you know the user simulator itself is any good, as distinct from the product agent it's testing?
By optimizing the simulator on three separate axes — coverage (entropy/distance metrics), realism, and cost — with the number of genuine agent failures it surfaces as the ultimate downstream success metric. - ✅ How do you keep a simulated-scenario test suite from going stale as real users and products change over time?
By continuously diffing production logs against the existing simulated distribution and synthesizing new scenarios to cover the gap. - ⭕ How do you validate soft, non-binary goals (e.g., 'give me a good product recommendation') that have no crisp completion condition?
The speaker proposes inferring acceptance/rejection from a model of the simulated user's own prior history, but this substitutes one model-generated proxy (the simulated user's modeled reaction) for another (the recommendation itself), without addressing how that proxy's own accuracy is validated. - ⭕ How do you avoid simulated personas collapsing into caricatures of the most salient or distinctive behaviors rather than faithfully representing real user diversity?
The speaker reframes the question as a cold-start (expert-interview-seeded) vs. warm-start (production-data-matched) distinction, but does not directly answer the caricature-vs-representativeness concern the question actually raised, especially for the harder cold-start case.
Gaps
- incompleteThe talk never explains how to validate that AI-simulated personas actually approximate real user intent/behavior distributions. Beyond appealing to domain-expert 'interviews' for cold start and matching production data for warm start, there is no calibration metric, statistical test, or audit process offered to confirm the synthetic distribution isn't itself biased — e.g., toward whatever a small number of experts (or the generating LLM) finds salient, which is exactly the caricature concern Participant 3 raised and that the speaker sidesteps rather than answers.
Read next: Sim-to-real transfer / distributional-fidelity literature (as used in robotics simulation), and work on eliciting representative rather than merely plausible personas from a small number of domain experts. - incompleteThe reliability of the LLM-as-judge itself — used for both the five-dimension per-turn scores and implicitly for the calibrated helpfulness/brand-voice metrics — is asserted, not demonstrated. There is no inter-rater agreement with human graders, no measurement of judge drift over time, and no discussion of whether the judge and the agent under test might share correlated blind spots if built on similar underlying models or seeded with examples from the same pipeline.
Read next: The corpus's own LLM-as-judge material — extract-infoq-strands-agents-clare-liguori#p5 (judge should be a different, smaller model than the agent under test) and extract-beyond-the-200-ok-observability-for-ai#g1 (what makes an automated judge reliable). - illogicalThe Q&A answer to the 'soft goals' question (e.g., recommending a product) defines evaluation success as whether a modeled version of the simulated user accepts or rejects the recommendation, based on that user's own prior history. This makes the ground truth itself the output of a model in the same simulation pipeline being evaluated — a circularity the speaker does not flag, since it means the 'ground truth' can never independently disconfirm the recommender it's judging.
Read next: Literature on reference-free/self-referential evaluation design and evaluator independence; the corpus's own material on judge-under-test correlation. - incompleteNo cost/ROI accounting is given for building and running the simulation infrastructure itself. The talk names running cost as a design constraint (avoiding, e.g., 10,000-conversation batches every CI run) but never quantifies the engineering cost of building personas/scenarios, judge-calibration labor, or how the total cost compares to the manual-tester process it replaces — beyond the qualitative, unquantified two-year Pearson relationship.
Read next: Corpus material on eval/observability infrastructure cost and token/infra economics, for a fuller before/after ROI picture of adopting simulation-driven testing.
Closes gaps elsewhere
- Beyond the 200 OK: Architecting Observability for AI · g1 — That document's evaluation-driven-development prescription presupposes a reliable automated judge for hallucination/faithfulness/relevance without saying what makes such a judge reliable. This talk gives a concrete recipe: ground the judge in a specific simulated scenario with a known, verifiable goal so task completion can be checked by rule-based verification against real backend state, and calibrate softer per-turn LLM-as-judge metrics (helpfulness, brand voice) with curated hard-negative and hard-positive examples.
- Don't train the model, evolve the harness · g2 — That gap flags the self-optimizing harness-evolution loop's total dependence on a reliable task judge with no guidance on how to build one (its own judge was a brittle exact-filename grader). This talk supplies the missing eval-construction methodology: build the judge around simulation-generated scenarios with automatically verifiable, rule-checkable goals rather than a single brittle exact-match grader.
Related documents
- Downstream #1: Where Do Our Tokens Go?
- Kaggle is making AI benchmark creation effortless
- The Anatomy of an Agent Harness
- 11 Tips For AI Coding With Ralph Wiggum
- 20 Loop Design Patterns Every AI Engineer Should Know
- Have your agent record video demos of its work with shot-scraper video
- Article: Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules
- [FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero
- Google's AlphaEvolve Reaches General Availability with Evolutionary Code Optimization as a Service
- Android Studio Quail 2 Redesigns Agent Mode, Streamlines AI-Assisted Coding