Calibrated Trust (in AI Agents)

The design goal of matching employee trust in an AI agent to its actual reliability — via explicit limitations, competence signaling, and preserved human control — rather than maximizing trust or defaulting to blanket skepticism.

Also known as: calibrated trust, appropriate trust, trust calibration

Part of: Trust

How authors use it

Oldest first (extraction date when no publication date is known).

Documents (7)

• To Adopt AI at Scale, Employees Need to Trust Agents Thomas McKinlay, Stefano Puntoni, Serkan Saka · 2026-09-09 source ↗ Reader 🖥️
  Because employees' willingness to grant AI agents real autonomy is governed by trust rather than technical capability, organizations can deliberately engineer calibrated (not blind) trust by being transparent about an agent's limitations, projecting competence over friendliness, tying its actions to users' broader goals, framing it as a subordinate helper rather than a powerful entity, and preserving human control over consequential decisions.
• NodeJS Proposes Built-In Virtual File System, Sparking Debate Over AI-Generated Contributions Daniel Curtis · 2026-05-25 source ↗ Reader 🖥️
  A proposal to add a built-in virtual file system to Node.js core — a large (~19,000-line) change that its author, TSC member Matteo Collina, built with heavy AI assistance while reviewing every line himself — has become a flashpoint for an unresolved governance question facing critical open-source infrastructure: whether AI-generated contributions should be accepted at all, given that reviewers cannot easily verify a human's claim to have adequately reviewed machine-generated code at that scale, even as the underlying feature draws strong independent technical support and has already been adopted by Vercel and LangChain.
• Gene Kim on Vibe Coding - Part II | Episode 411 | Troubleshooting Agile Douglas Squirrel · 2025-12-18 source ↗ Reader 🖥️
  Vibe coding changes software teams' organizational unit of work — toward smaller, option-rich teams and even individual business-stakeholder-plus-developer pairs rather than net headcount reduction — and because trust in these tools is a learned skill that takes about a year to build, leaders should start adopting now rather than wait for the technology to mature further.
• Coding agents have crossed a chasm singleton.io · 2025-06-14 source ↗ Reader 🖥️
  Autonomous coding agents have crossed a real capability threshold -- from 'smarter autocomplete' and pair-programming assistance to genuine delegate-to collaborators the author can hand whole tasks to -- which shifts the developer's bottleneck from writing code to framing problems well enough for an agent to reason about them, while the developer's own technical foundation and review discipline remain what separates good AI-assisted outcomes from confidently-wrong ones.
• Practices and Challenges of Using GitHub Copilot: An Empirical Study Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, Muhammad Waseem · 2023-03-15 source ↗ Reader 🖥️
  Mining Stack Overflow and GitHub Discussions from Copilot's first ~17 months shows that the tool's cited benefits and limitations are near-mirror categories of each other (useful code generation vs. limited/poor-quality generation, better vs. unfriendly user experience), making Copilot adoption a case-by-case trade-off across IDE choice, budget, privacy tolerance and code quality rather than a simple win.
• Papers on the UX of AI programming assistants Austin Z. Henley · 2023-02-20 source ↗ Reader 🖥️
  Across the first wave of 2022-23 HCI/SE studies on AI programming assistants (chiefly GitHub Copilot and OpenAI Codex), the user-experience bottleneck is not the AI's ability to generate plausible code but the human's ability to discover, understand, trust, and recover from what it generates — shifting the programmer's core activity from writing code toward assessing it.
• Study finds AI assistants help developers produce code that's more likely to be buggy Thomas Claburn · 2022-12-13 Reader 🖥️
  A controlled Stanford user study found that developers given access to an AI coding assistant (OpenAI's codex-davinci-002, as used in GitHub Copilot) produced measurably less secure code than a control group working without it — and, more strikingly, were more likely to believe their own AI-assisted code was secure than the control group was about theirs — indicating AI assistance can degrade both code security and developers' self-assessment of that security simultaneously.

Propositions touching this concept (2)

By publication month, oldest first (extraction month when no publication date is known).

2026-05

• explicitThe appropriate trust signal for software is no longer the quality of its code, tests, or documentation, but whether someone has actually used it over a sustained period.
  Vibe coding and agentic engineering are getting closer than I'd like · p2

2026-09

• explicitdevelopedThe design goal is calibrated trust — employees understanding where an agent is capable, where it will likely fail, and when they remain in control — not blind or maximal trust.
  To Adopt AI at Scale, Employees Need to Trust Agents · p12

Open gaps (7)

• incompleteThe article presents only two framings — tool vs. employee — without exploring intermediate metaphors (e.g., 'assistant', 'contractor', 'infrastructure'). The design space between 'tool' and 'colleague' is unexplored, yet organizational reality may require something in between.
  Research: Why You Shouldn't Treat AI Agents Like Employees — read next: Research on human-automation interaction taxonomies; Parasuraman & Riley on levels of automation and trust calibration
• uninformedThe '12 months to build trust in AI, plateauing at 15 months' finding is presented as coming from Kim's 2025 DORA collaboration but explicitly 'didn't make it into the report' — no sample size, measurement method, or population is given, so the claim can't be checked.
  Gene Kim on Vibe Coding - Part II | Episode 411 | Troubleshooting Agile — read next: Look for a published version of this finding, or a primary interview with Kim/DORA researchers describing the methodology.
• incompleteThe list has no stated inclusion criteria, methodology, or search strategy — it is simply papers the author personally came across, including one he co-authored (the IntelliCode study), with no discussion of how representative the dozen papers are of the broader 2022-23 literature or what populations/tools are missing (longitudinal studies, non-Copilot tools, non-English-speaking programmer populations).
  Papers on the UX of AI programming assistants — read next: A later systematic review or meta-analysis of AI-coding-assistant UX studies, and the corpus's own later empirical pieces for how these early qualitative findings held up once tooling matured.
• incompleteNo information is given on the actual correctness or quality of the generated node:vfs implementation itself (test coverage, bug reports, independent security review) — the entire debate as reported is about process and authorship, not about whether the 19,000 lines are good code.
  NodeJS Proposes Built-In Virtual File System, Sparking Debate Over AI-Generated Contributions — read next: Follow the PR (nodejs/node #61478) for review comments and any post-merge incident reports once the TSC policy vote resolves.
• uninformedThe piece does not report what specific policy options the TSC is actually considering (outright ban, disclosure requirement, scale threshold, mandatory second reviewer, etc.), so the 'pending vote' framing has no visible menu of outcomes to evaluate against the two positions it does report.
  NodeJS Proposes Built-In Virtual File System, Sparking Debate Over AI-Generated Contributions — read next: Track the Node.js TSC's published AI-contribution policy once adopted, and compare it against the positions staked out by Indutny's petition and its critics.
• incompleteThe study measures self-reported confidence immediately after writing code in a lab setting; it does not examine whether this overconfidence propagates into real-world consequences — e.g. whether overconfident developers skip code review, merge faster, or escalate less — which is the actual mechanism by which miscalibrated trust would cause harm in production teams.
  Study finds AI assistants help developers produce code that's more likely to be buggy — read next: extract-to-adopt-ai-at-scale-employees-need-to-trust-agents takes up trust miscalibration (both deficit and excess) as a general organizational-adoption problem and would be the natural next read for the downstream consequences this study doesn't measure.
• incompleteThe method mines self-selected public posts (people who chose to complain or praise on Stack Overflow/GitHub Discussions), not a representative sample or controlled study of all Copilot usage — the authors name this as an external-validity threat but do not quantify how it might bias the reported percentages (e.g., toward louder complaints or toward technically engaged early adopters).
  Practices and Challenges of Using GitHub Copilot: An Empirical Study — read next: Controlled or survey-based Copilot usage studies that sample practitioners directly rather than mining public complaint/praise text.