Synthetic User Simulation
Using AI agents to simulate diverse end-user personas — with goals seeded from a target agent's own capabilities — that converse with a product agent to generate realistic multi-turn trajectories, used to test, evaluate, and (in privacy-restricted domains) train other agents before real users interact with them.
Also known as: synthetic users, simulated users, user simulation, trajectory entropy, tool call distribution entropy, tool call transition entropy, arksim, golden scenario set
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation Zhou Yu · 2026-09-07 | Trajectory entropy (tool call distribution entropy / tool call transition entropy / trajectory distance) | A family of quantitative diversity metrics for the *simulator*, not the product agent: how spread out the set of tools a synthetic user invokes is (distribution entropy), how varied the sequence in which it moves between tools is (transition entropy), and how far apart different simulated conversations are from each other (trajectory distance) — together used to detect and penalize a simulator that keeps generating near-identical, low-coverage test conversations. |
| Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation Zhou Yu · 2026-09-07 | Golden scenario set (as 'ever-evolving') | A curated, versioned collection of (profile, goal, context) test scenarios treated as a living CI/CD regression suite that is deliberately and continuously revised — new scenarios are synthesized whenever production logs reveal a gap between real and simulated user distributions. |
Documents (1)
- Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation Zhou Yu · 2026-09-07 source ↗ Reader 🖥️
Because static, single-turn benchmarks with fixed golden answers cannot evaluate multi-turn, tool-using agents whose correctness depends on live backend state and conversational history, teams must replace ad hoc manual testing with AI-simulated user personas that generate realistic, coverage-optimized conversation trajectories against which scenario-grounded, automated evaluation can run inside CI/CD — turning agent testing from the manual bottleneck that keeps roughly 95% of agents stuck in demo into a scalable, continuously-refreshed, self-improving production pipeline.
Propositions touching this concept (1)
By publication month, oldest first (extraction month when no publication date is known).
2026-09
- implicitSynthetic user simulation is valuable not only for testing but as a training-data-generation mechanism for domains where real data either doesn't exist yet (a brand-new agent, cold start) or is legally unusable (PII-restricted domains like mortgage underwriting).
Open gaps (2)
- incompleteThe talk never explains how to validate that AI-simulated personas actually approximate real user intent/behavior distributions. Beyond appealing to domain-expert 'interviews' for cold start and matching production data for warm start, there is no calibration metric, statistical test, or audit process offered to confirm the synthetic distribution isn't itself biased — e.g., toward whatever a small number of experts (or the generating LLM) finds salient, which is exactly the caricature concern Participant 3 raised and that the speaker sidesteps rather than answers.Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation — read next: Sim-to-real transfer / distributional-fidelity literature (as used in robotics simulation), and work on eliciting representative rather than merely plausible personas from a small number of domain experts.
- incompleteNo cost/ROI accounting is given for building and running the simulation infrastructure itself. The talk names running cost as a design constraint (avoiding, e.g., 10,000-conversation batches every CI run) but never quantifies the engineering cost of building personas/scenarios, judge-calibration labor, or how the total cost compares to the manual-tester process it replaces — beyond the qualitative, unquantified two-year Pearson relationship.Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation — read next: Corpus material on eval/observability infrastructure cost and token/infra economics, for a fuller before/after ROI picture of adopting simulation-driven testing.