Papers on the UX of AI programming assistants

Austin Z. Henley · article · 2023-02-20 source ↗ Reader 🖥️

Calibrated Trust (in AI Agents)Code Was Never the Bottleneck Coding Agents Human as Orchestrator of Agents

Across the first wave of 2022-23 HCI/SE studies on AI programming assistants (chiefly GitHub Copilot and OpenAI Codex), the user-experience bottleneck is not the AI's ability to generate plausible code but the human's ability to discover, understand, trust, and recover from what it generates — shifting the programmer's core activity from writing code toward assessing it.

theoretical · expository — A curated, annotated bibliography — Henley doesn't argue directly, but the selection and one-sentence (ChatGPT-generated) summaries tacitly assert a thesis about what this early wave of research collectively found.

Structure

Part	Role
Stated motivation: the volume of UX research on AI coding assistants had become too large to track informally	context
Twelve annotated papers (CHI, UIST, OOPSLA, ICSE, FSE, IUI, PPIG, ACM Queue, two arXiv preprints), each reduced to authors, a one-sentence ChatGPT-written finding summary, and venue	evidence
Open invitation to submit further papers via GitHub PR	context

Key terms

Bimodal interaction
From the Barke/James/Polikarpova paper this list highlights: programmer interaction with Copilot-style suggestions falls into two distinct modes rather than a smooth continuum — an acceleration mode (speeding up code the programmer already knows how to write) and an exploration mode (using the tool to discover unfamiliar APIs or approaches).

Propositions

explicitResearch on AI programming-assistant UX surged sharply once GitHub Copilot launched, enough that keeping up with the literature already required a dedicated, continuously-updated curated list by early 2023.conf: high
explicitProgrammer interaction with Copilot-style code suggestions is bimodal rather than a smooth continuum: programmers either accelerate code they already know how to write, or explore unfamiliar approaches/APIs — two distinct usage patterns that call for different interface support.conf: medium
explicitAI code-recommendation tools are underused in practice chiefly because of poor discoverability, not because developers reject the quality of the suggestions — design principles and new inline interfaces specifically targeting discoverability are proposed to increase usage.conf: medium
explicitDevelopers prefer Copilot-style generation for everyday programming tasks, but understanding, editing, and debugging the generated code — not writing the initial prompt — is what actually limits task-solving effectiveness.conf: medium
explicitAI code generation significantly increases code-authoring performance for novice programmers without harming their performance on manual code-modification tasks, and novices with stronger prior knowledge retain more from having used it.conf: medium
explicitAs AI coding tools get integrated into more parts of the development workflow (code review, defect fixes, test writing), developers' work shifts toward spending more time assessing AI suggestions relative to the task than doing the task themselves.conf: high
supportsThe New SDLC With Vibe Coding: From ad-hoc prompting to Agentic Engineering · p14 — Bird et al.'s early-2023 claim that AI integration shifts developer work from doing the task to assessing suggestions anticipates the later, empirically-backed (METR study) finding that AI assistance relocates work from writing to reviewing/verifying.
explicitConversational, multi-turn interaction grounded directly in the code under discussion (not just one-shot suggestion) is a viable interface style for an AI programming assistant with potential productivity benefits beyond single-line completion.conf: medium
explicitThe input parameters given to a code-generating model — task description, surrounding context, sampling 'creativity'/temperature, and number of candidate solutions — materially affect output quality, suggesting room for automated strategies to tune these rather than leaving them to ad hoc user choice.conf: medium
explicitLLM-assisted programming is a genuinely new way of programming with its own distinct properties and challenges, not simply a faster version of earlier programmer-assistance paradigms (autocomplete, pair programming, program synthesis).conf: medium

Arguments

The central challenge of AI-assisted programming, as of early 2023, had already moved from code-generation quality to human-AI interaction design. strength: low

Explicit premises:

Multiple independently-run studies (discoverability, bimodal use, understanding/editing/debugging, task shift toward assessment) each isolate a different interaction bottleneck rather than a generation-quality bottleneck
Henley curated these specifically as UX papers, implicitly separating them from capability/benchmark research on the models themselves

Implicit premises:

A one-sentence, ChatGPT-generated summary of each abstract reliably captures a paper's actual finding well enough to ground a claim about the field's center of gravity
The twelve selected papers are a representative sample of the broader 2022-23 literature rather than a convenience sample assembled from whatever the author personally came across — including one paper he co-authored (the IntelliCode study)

Inferences:

If representative, this reframes 'AI coding assistant adoption' as primarily a UX/interaction-design problem rather than a raw model-capability problem — a reframing that later industry narratives centered on ever-larger, more capable models implicitly push back against

Questions addressed

✅ Given that AI-generated code is usually plausible-looking, what specifically limits its usefulness in practice?
Not generation quality itself, but the feature's discoverability, and the human effort required afterward to understand, edit, and debug what gets generated.

Gaps

incompleteThe list has no stated inclusion criteria, methodology, or search strategy — it is simply papers the author personally came across, including one he co-authored (the IntelliCode study), with no discussion of how representative the dozen papers are of the broader 2022-23 literature or what populations/tools are missing (longitudinal studies, non-Copilot tools, non-English-speaking programmer populations).
Read next: A later systematic review or meta-analysis of AI-coding-assistant UX studies, and the corpus's own later empirical pieces for how these early qualitative findings held up once tooling matured.
incompleteEvery study here concerns single-shot or short-turn code suggestion/completion (Copilot, Codex, IntelliCode); none anticipates the fully agentic, multi-step, tool-using coding agents (editing files across a repo, running tests, opening pull requests) that much of the later corpus treats as the default interaction model — so findings about discoverability and bimodal acceleration/exploration use may not transfer to a fundamentally different interface.
Read next: extract-kniberg-future-of-work-jfokus and other corpus pieces on fully agentic coding tools, to check whether 'discoverability' and 'bimodal' use resurface as UX concerns in an agentic interface or get superseded by new problems (reviewing diffs, trusting autonomous PR creation).