How Microsoft Developers Use AI in Real-World Coding | BRK103
Microsoft Developer · video · 2025-05-21 source ↗ Reader 🖥️
Agentic EngineeringCoding AgentsCognitive DebtHuman-in-the-LoopLegacy & Organizational ForgettingVibe Coding
AI tools are most valuable to an individual developer not as autocomplete or as a replacement for judgment, but as a fast, tireless conversational partner for ideation, documentation, test generation, and debugging across a developer's own long-lived codebase — provided a knowledgeable human stays in the loop to evaluate, correct, and hold accountability for what the AI produces before it reaches the team's review process.
practical · narrative — A conference talk by two .NET-team developers (Steven Tobe, David Felder) walking through their own personal workflows for using AI tools (ChatGPT, Copilot chat/completions/agent mode) across ideation, documentation, code generation, testing and debugging, illustrated with real PRs from the .NET and MCP C# SDK repos, including one live on-stage demo of Copilot agent mode.
Structure
| Part | Role |
|---|---|
| Framing / disclaimers (opening) | context |
| Ideation: bouncing ideas off AI on a 10-year-old memory-pool bug and a regex-engine optimization | evidence |
| Maintainability: AI-generated documentation for a cryptic 2003 regex interpreter and the MCP C# SDK | evidence |
| Code generation: LINQ Shuffle, memory-extensions overloads, reservoir sampling, hypergeometric-distribution optimization | evidence |
| Testing: AI-written edge-case test suites, including tests a human reviewer said they would never have written | evidence |
| Debugging: root-causing a production memory leak from a bug-report email in ~45 seconds | evidence |
| Live demo of GitHub Copilot agent mode fixing an Aspire repo issue end-to-end on stage | evidence |
| Closing: 'explore, try new things,' tools change weekly | conclusion |
Key terms
- vibe coding Vibe Coding
Letting an AI agent write code end-to-end without reading or understanding it yourself, acceptable only for throwaway, low-longevity tools (e.g. the presenter's 8-minute hotel-room game); explicitly not how either presenter works on code that has to be maintained and tested. - ideation (with AI)
A deliberate conversational mode where the goal is explicitly not to get 'the answer' or a quick fix, but to generate multiple competing approaches rapidly (e.g. 'give me four approaches, don't give me the answer') so the human can reason about trade-offs before any code is written. - it will glaze you
The AI's tendency toward sycophantic agreement with the user's framing ('You are correct, David. You are so smart for saying that'), named explicitly as a known failure mode to watch for, not a sign the AI has actually validated the idea. - orchestrate the agents
A half-joking reframing of the developer's remaining job — once AI increasingly handles ideation, documentation, code and tests, the human's distinctive contribution becomes directing, reviewing, and being accountable for multiple AI-generated artifacts rather than typing code.
Propositions
- explicitFor code with longevity (needs maintenance and testing), AI is used primarily for ideation, documentation, test generation and debugging assistance — not for end-to-end 'vibe coded' authorship, which both presenters reserve for throwaway tools.conf: high
- explicitAsking an AI for multiple competing approaches rather than a single answer ('give me four approaches, don't give me the answer') produces more useful ideation than asking it to just solve the problem.conf: high
- explicitAI-generated prototypes required substantial additional engineering by a dedicated engineer — performance testing, added metrics, trade-off evaluation — before being accepted as production code; the AI's output was a seed for the conversation, not the shipped implementation.conf: high
- explicitAI is effective at reverse-engineering and documenting old, cryptic, unowned code — generating accurate explanatory comments for a 2003 regex interpreter whose original author was no longer at the company and whose five-year-old TODO comment had never been acted on.conf: highextendsHarness engineering for coding agent users · p14 — Bockeler argues legacy/high-entropy codebases are where a test harness is most needed but hardest to build; this talk's regex-interpreter and memory-pool examples show AI providing a lighter-weight partial workaround — recovering comprehension and generating documentation/tests for orphaned legacy code even without a full harness.
- explicitA developer can use an LLM to script bulk changes across an entire codebase (6,000 lines of AI-generated XML doc comments for the MCP C# SDK) faster than writing them by hand, but a senior engineer still needed to manually review and cut the output by roughly a third because much of it was 'correct but over the top.'conf: high
- explicitAI suggestions can introduce genuinely new, correct domain knowledge outside the developer's own expertise (reservoir sampling, hypergeometric distribution for a probability edge case), making AI valuable as a knowledge amplifier, not only a typing-speed amplifier.conf: high
- explicitAI-generated code and regex patterns are not reliably correct against formal specifications and require a human who knows the domain to catch errors (e.g. an AI-generated URI-template regex that allowed unlimited digits when the RFC specifies exactly four, and incorrect percent-encoding handling).conf: high
- explicitAI can root-cause a subtle production bug from an informal bug report (a copy-pasted customer email) far faster than a human reading through a memory dump cold — in this case, in about 45 seconds — though it still proposed some incorrect fixes alongside the correct one.conf: highsupportsPractices and Challenges of Using GitHub Copilot: An Empirical Study · p4 — Zhang et al.'s survey finds 'poor quality of generated code' and integration friction as leading cited Copilot limitations; this talk's own account (AI proposing 3 of 4 wrong fixes for the pipe leak, an over-broad regex, incorrect percent-encoding) is a first-person confirmation of the same unreliability pattern from presenters who are otherwise AI enthusiasts.
- explicitAgent mode's main practical advance over chat/completion AI is that it searches the codebase itself to establish context, reducing (but not eliminating) the amount of explicit file/folder context a developer must supply, especially once there is prior conversation history to draw on.conf: high
- implicitEven in a live, low-stakes on-stage agent-mode demo, the presenters accepted the agent's output (including a test that did not compile) largely without independent verification, played this as a joke about accountability ('if it's bad, it's your fault'), and still merged the result.conf: medium
- explicitMatching the model to the task (a 'deep thought' model for complex ideation vs. a faster/cheaper model for a simple, well-scoped coding task) is treated as its own learned skill rather than a fixed rule.conf: medium
Arguments
AI's biggest value for an individual senior engineer is accelerating and widening ideation (exploring many approaches cheaply) rather than producing final code. strength: medium
Explicit premises:
- Going from idea to real production software is normally slow and effortful (whiteboard sketch at best from an hour-long meeting; hours of solo coding for a first buggy implementation).
- An AI chat can produce four distinct implementable approaches in seconds.
- The presenter explicitly does not want 'the answer' from these sessions, only expanded option space.
Implicit premises:
- The bottleneck in solving hard, long-deferred problems is more often the cost of generating and comparing candidate approaches than the cost of typing code.
- A senior engineer retains the judgment to select and adapt among AI-proposed approaches; the technique would be less safe in less experienced hands.
Inferences:
- If ideation-acceleration is the main value, then AI's impact on a codebase scales with how experienced/judgment-equipped the human using it already is — it is an amplifier of existing expertise, not a substitute for it.
As AI increasingly absorbs the 'toil' layer of software work (docs, test enumeration, string-escaping edge cases, bulk mechanical changes), the distinctive remaining human job shifts toward directing, scoping, and being accountable for AI output — 'orchestrating the agents' — rather than typing most of the code. strength: medium
Explicit premises:
- Several categories of work the presenters dislike doing themselves (doc comments, enumerating test edge cases, writing shell-escaping code) are now delegated wholesale to AI.
- The presenters explicitly frame remaining human involvement as review, direction, and blame/credit allocation ('orchestrate the agents', 'you can blame it when things go wrong').
Implicit premises:
- Toil removal is unambiguously a net good for the organization, independent of whether removing the toil also removes the slow, manual exposure to the codebase that built the humans' own current expertise.
- Accountability ('it's your fault if it's bad') is sufficient incentive to keep humans rigorously checking AI output, even when demo conditions (stage pressure, time limits) visibly push toward skipping verification.
Inferences:
- If toil-removal continues at this pace for junior engineers (not just the senior presenters), the manual-toil pathway by which today's senior engineers built their own deep codebase familiarity may not be available to the next generation in the same form — a generational knowledge-transfer question the talk raises only as a joke, never examines.
Narratives
- A 10-year-old out-of-memory bug in Azure App Service's connection memory pool, long punted as too hard, gets a fix after the presenter uses ChatGPT/Copilot to generate four candidate mitigation strategies in seconds, then works with an engineer who hardens one into a production, metrics-instrumented implementation.
→ Supports the proposition that AI's value is ideation-acceleration on long-deferred problems, and that AI prototypes still require full engineering rigor before shipping. - AI generates accurate explanatory XML comments for a ~3,000-line, undocumented 2003 regex interpreter whose original author is gone and whose own five-year-old 'figure this out later' comment had never been resolved.
→ Supports the claim that AI can recover understanding from orphaned legacy code — a concrete instance of organizational forgetting being partially reversed by AI, not just accelerated by it. - A developer scripts an LLM to bulk-generate 6,000 lines of XML doc comments across the entire MCP C# SDK codebase in one pass; a colleague then manually trims and corrects about a third of it before merging.
→ Illustrates both the scale-advantage of AI for mechanical documentation work and the continued necessity of expert human editing to avoid shipping verbose, 'correct but over the top' output. - A bug-report email from a partner team describing an intermittent memory leak is pasted directly into an AI chat, which root-causes the actual defect (an uncleaned pending read operation on pipe disposal) in about 45 seconds, though three of its four proposed fixes were wrong.
→ Supports the proposition that AI dramatically compresses the 'orientation' phase of debugging (understanding what's wrong) even when it does not reliably produce the correct fix on its own. - Live on stage, Copilot's coding agent is assigned an Aspire-repo issue in a fresh GitHub Codespace; it searches the codebase, proposes scope, edits 66 resource files with required doc comments, and (after being asked) adds a non-compiling test — all while the presenters repeatedly say 'yes' without independent verification and then merge.
→ Functions as an unintentional counter-example to the talk's own stated norm of human rigor: under live-demo conditions, verification was skipped even though the presenters had just spent the whole talk emphasizing the need for it on their day-to-day (non-demo) work.
Questions addressed
- ✅ What does disciplined, senior-engineer-level AI usage on a long-lived, maintained codebase actually look like, day to day, beyond headline 'vibe coding' narratives?
A five-category taxonomy from the presenters' own practice: ideation, documentation/maintainability, code generation, testing, and debugging — each paired with an explicit account of where human review stayed necessary. - ⭕ Does agent-mode autonomy reduce the human verification the presenters say is necessary everywhere else in the talk?
Gaps
- incompleteEvery example is this talk's two presenters narrating their own successes on their own codebase; there is no instance of an AI-assisted change that was rejected, reverted, or caused an incident, and no outside voice (a reviewer who pushed back, a postmortem) is included — survivorship bias in the evidence base.
Read next: Practitioner retrospectives that include AI-assisted failures or reverted PRs, not just successes. - illogicalThe talk spends most of its runtime establishing that AI output needs careful human verification (checking specs, catching sycophancy, hardening prototypes into production code), then in the live demo repeatedly accepts agent output — including a non-compiling test — without the verification it just finished recommending, and frames this inconsistency as a joke about blame rather than addressing it.
Read next: Material on verification discipline under time/social pressure (demo conditions, deadline pressure) versus stated best practice — the gap between espoused and enacted review rigor. - incompleteThe talk is explicitly individual-practitioner-scoped ('we are speaking only for the two of us') and never addresses how these personal practices interact with team-level process: how AI-assisted PRs get reviewed by colleagues who were not part of the AI conversation, whether reviewers can tell how much of a PR was AI-authored, or whether the team has any shared norms about when AI ideation is disclosed versus just presented as the engineer's own reasoning.
Read next: Team-level studies of how AI-authored or AI-assisted contributions are reviewed, disclosed, and normed within a team, as distinct from individual productivity.
Closes gaps elsewhere
- Charlie Hills 🦩's Post · g3 — Provides a concrete, detailed counter-case of solo (not team), legacy-codebase AI usage — showing the practice differs sharply from greenfield 'vibe coding': ideation-and-verify rather than autonomous authorship — directly illustrating the context-dependence this gap says is missing, albeit for Copilot/ChatGPT rather than Claude Code specifically.
- Claude Code: Best practices for agentic coding · g6 — Same partial closure as above: a first-person, solo-developer, legacy-codebase account of how AI-assisted practice actually varies by context (throwaway vs. long-lived code), though still not team-based and not Claude-Code-specific.
Related documents
- Practices and Challenges of Using GitHub Copilot: An Empirical Study
- Have your agent record video demos of its work with shot-scraper video
- [FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero
- 11) AI-spesial med Vilde Opsal og Anders Sveen
- Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE
- Looking back at the Future of Software Engineering unconference
- The New SDLC With Vibe Coding: From ad-hoc prompting to Agentic Engineering
- Coding agents have crossed a chasm
- Why did Martin Fowler invite 60 engineers to a ski resort?
- Vibe coding and agentic engineering are getting closer than I'd like