Practices and Challenges of Using GitHub Copilot: An Empirical Study

Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, Muhammad Waseem · pdf · 2023-03-15 source ↗ Reader 🖥️

theoretical · mixed — Primarily a descriptive empirical study (mining Stack Overflow posts and GitHub Discussions for categorized findings) building toward factual claims about how practitioners used early Copilot — but closes with a practical implications section recommending mainstream-IDE adoption and trade-off awareness.

Structure

Key terms

• Double-edged sword
  The authors' own framing for the finding that Copilot's benefit categories and limitation categories mirror each other one-for-one (e.g. useful code generation vs. limitation to code generation), so there is no single verdict on the tool — only situational trade-offs.
• Difficulty of integration
  The single most-cited limitation (28.0% of limitation mentions): Copilot failing to work with, or conflicting with, a developer's chosen IDE or other installed plug-ins — distinct from server/account access problems or code-quality problems.

Propositions

• explicitThe most-discussed programming languages used with Copilot are JavaScript (20.7%) and Python (20.1%), with Visual Studio Code the dominant IDE (46.0%) and Node.js the leading technology (41.7%) among discussions mentioning a specific technology.conf: high
• explicitThe leading function category practitioners report implementing via Copilot is data processing (26.7%), followed by front-end element control, string processing, and test writing (11.1% each).conf: high
• explicitThe most-cited benefit of using Copilot is useful code generation (49.0% of benefit mentions), followed by faster development (16.3%) and better code quality (10.2%).conf: high
• explicitcorroboratedThe most-cited limitation is difficulty of integration with IDEs/plug-ins (28.0% of limitation mentions), followed by difficulty accessing Copilot's servers (17.5%), a hard limit on generated-response length (14.6%), and poor quality of generated code (11.6%).conf: high
  supportsI'm done using AI · p1 — This study's systematically coded 2023 limitation categories (difficulty of integration, poor/limited code generation) are the empirically-grounded, contemporaneous counterpart to Brett's informal 2026 recollection that early Copilot autocomplete was 'distracting and basically useless.'
  supportsMartin Fowler & Kent Beck: Frameworks for reinventing software, again and again · p15 — Provides the dated, documented evidence base for the premise that genuine early negative impressions of Copilot-era autocomplete existed and were widespread — the claim that proposition is explicitly reasoning about.
  supports ←How Microsoft Developers Use AI in Real-World Coding | BRK103 · p8 — Zhang et al.'s survey finds 'poor quality of generated code' and integration friction as leading cited Copilot limitations; this talk's own account (AI proposing 3 of 4 wrong fixes for the pipe leak, an over-broad regex, incorrect percent-encoding) is a first-person confirmation of the same unreliability pattern from presenters who are otherwise AI enthusiasts.
• explicitCopilot's benefit and limitation categories are near mirror images of each other (useful code generation vs. limitation to code generation; better code quality vs. poor quality of generated code; better user experience vs. unfriendly user experience), which is the evidentiary basis for the paper's 'double-edged sword' conclusion.conf: high
  supportsKI, utvikling og følelser · p15 — The mirrored benefit/limitation categories and quoted mixed verdicts from 2022-23 are an early data point in the same first-Copilot-wave cohort experience that proposition says produced developer fatigue before more capable models arrived.
• explicitPractitioners cite code privacy as a distinct concern from code quality — worrying that Copilot may use their private-repository code without permission (7.5% of limitation mentions).conf: high
• explicitPractitioner reports describe the quality of Copilot's suggestions degrading ('becomes unacceptable') as the files being worked on get larger.conf: medium
• explicitChoosing a mainstream IDE (Visual Studio Code, Visual Studio, IntelliJ IDEA, NeoVim, PyCharm — together 85.9% of IDE mentions) correlates with smoother Copilot integration, while choosing a less-common IDE correlates with reported integration difficulty.conf: medium
• explicitAs of this study (data collected through November 2022), Copilot was available only on individual accounts, with no organization-level edition for teams to purchase or manage centrally.conf: high

Arguments

Narratives

• Quoted developer reactions pull in opposite directions on the same axis: one says Copilot 'is a lot more fun to use and does not annoy me like some other AI systems' (GitHub #7254) while another calls it 'an amazing tool with poor user experience' (GitHub #8468).
  → Illustrates the double-edged-sword thesis concretely — the same tool producing opposite verbatim verdicts from different practitioners.

Questions addressed

• ✅ What languages, IDEs, and technologies do practitioners actually use with Copilot?
  Answered via descriptive statistics over 169 Stack Overflow posts and 655 GitHub Discussions.
• ✅ What benefits and limitations do practitioners report when using Copilot?
  Answered via qualitative constant-comparison coding into 10 benefit and 15 limitation categories.
• ⭕ Under what conditions do Copilot's challenges manifest as advantages versus disadvantages, and how can practitioners convert the latter into the former?
  Explicitly named as future work ('Towards an Effective Use of Copilot') rather than answered here.
• ⭕ Who uses Copilot (developers vs. educators vs. students), and when/how/for what purposes?
  Explicitly acknowledged as unexamined by this study and left for future work.

Gaps

• incompleteThe data window (Copilot's launch in June 2021 through the November 2022 search date) predates Copilot Chat, inline-chat editing, the @workspace context tag, and Copilot Enterprise (all added by early-to-mid 2024, per later Radar coverage) — so categories like 'difficulty of integration' and 'limitation to code generation' (a ~1000-character response cap) describe the autocomplete-only era specifically and may already have been substantially addressed by the time the paper published.
  Read next: Later, chat-era Copilot studies or Radar coverage to see which of these 15 limitation categories persisted versus were resolved by subsequent feature releases.
• incompleteThe method mines self-selected public posts (people who chose to complain or praise on Stack Overflow/GitHub Discussions), not a representative sample or controlled study of all Copilot usage — the authors name this as an external-validity threat but do not quantify how it might bias the reported percentages (e.g., toward louder complaints or toward technically engaged early adopters).
  Read next: Controlled or survey-based Copilot usage studies that sample practitioners directly rather than mining public complaint/praise text.

Related documents

• KI, utvikling og følelser
• Martin Fowler & Kent Beck: Frameworks for reinventing software, again and again
• I'm done using AI
• GitHub Copilot CLI combines model families for a second opinion
  Coding Agents LLM Reliability Failures
• Papers on the UX of AI programming assistants
  Calibrated Trust (in AI Agents) Coding Agents
• Study finds AI assistants help developers produce code that's more likely to be buggy
  Calibrated Trust (in AI Agents) LLM Reliability Failures
• Coding agents have crossed a chasm
  Calibrated Trust (in AI Agents) Coding Agents