Presentation: Leadership in AI-Assisted Engineering

Justin Reock · rss · 2026-04-20 Reader 🖥️

Code Was Never the Bottleneck DORA & Engineering Metrics Enterprise AI Adoption & Readiness Measuring AI Impact Psychological Safety

Engineering leaders who frame AI as a throughput multiplier rather than a cost-cutting tool — and back that framing with clear policies, learning time, psychological safety, oppositional measurement, and SDLC-wide integration — are the ones whose organizations see positive ROI from AI adoption, while top-down utilization mandates reliably backfire.

practical · mixed — Conference presentation transcript combining empirical data (DX/DORA research) with prescriptive leadership guidance. Argumentative in structure but delivered as a practitioner talk with case studies and Q&A. Ultimately argues for specific leadership actions.

Structure

Part	Role
Current state of AI impact — conflicting signals	Establishes the problem: aggregate data looks modestly positive, but per-company variance is enormous, so averages mislead
The learning curve and why mandates fail	Evidence that initial adoption causes productivity dips and that top-down mandates decrease psychological safety
DORA Bayesian analysis of effective strategies	Empirical backing for the specific leadership levers: clear policies (90%), time to learn, alleviating displacement worries (80%)
Distinct leadership strategies (six pillars)	Core prescriptive framework: integrate across SDLC, unblock usage, measure with oppositional metrics, reduce fear, build compliance/trust, tie to employee success
Reducing fear of AI — SWE-bench and MIT NANDA data	Provides leaders ammunition to credibly reassure engineers: AI completes only 44% of tasks autonomously, 95% of AI pilots fail, plateau in capability gains
Transparency, Goodhart's Law, and the Cobra Effect	Warns against metric-as-target thinking; makes the case for open communication about why metrics are collected
DX AI Measurement Framework	Provides concrete measurement guidance: utilization → impact → cost as a maturity curve, oppositional metrics of speed vs. quality
Compliance and trust — system prompts and temperature	Technical levers for output quality: feedback loops on system prompts, determinism via temperature control
Employee success and the DX GenAI study	Empirical evidence for high-value use cases (stack trace analysis #1) and onboarding acceleration (time to 10th PR halved)
SDLC integration case studies	Real-world examples (Morgan Stanley, Zapier, Canva, Faire, Spotify) showing AI applied beyond code generation to the actual bottlenecks

Key terms

throughput Measuring AI Impact
Total value delivered by the engineering organization to customers; the constraint that matters most per Theory of Constraints. Reock explicitly contrasts this with 'cost' as the wrong optimization target.
oppositional metrics Measuring AI Impact
Metrics deliberately placed in tension with each other (e.g., speed vs. quality) so that gaming one metric surfaces degradation in another. Inherited from the SPACE framework's insistence on multi-dimensional measurement.
change confidence DORA & Engineering Metrics
A qualitative DX metric measuring how confident an engineer feels that a change will not break production, scored via top-box Likert
Core 4 DORA & Engineering Metrics
DX's distilled measurement framework combining DORA, SPACE, and DevEx into four core metrics. Positioned as the foundation that AI metrics should layer onto, not replace.
temperature
The randomness parameter in LLM token selection; Reock frames it as a compliance lever leaders can use to control output determinism
time to 10th PR Measuring AI Impact
Onboarding metric: calendar time from a new engineer joining to their 10th accepted, merged pull request. DX reports AI-assisted organizations halve this.

Propositions

explicitAverage AI impact metrics hide enormous per-company variance, making aggregate data misleading for decision-makingconf: high
refinesMeasuring AI impact, assessing readiness, and new data trends · p5 — Qualifies the 10-15% average throughput gain by showing aggregate AI-impact figures hide enormous per-company variance, making the average misleading for decisions.
explicitThere is a distinct, universal learning curve where initial AI adoption causes productivity and quality to decline before recoveringconf: high
explicitcorroboratedTop-down AI utilization mandates decrease psychological safety and lead to metric gaming rather than genuine productivity gainsconf: high
supports ←Engineering Leadership Report 2026 · p16 — The free-text respondent describing a mandated AI tool plus an internal token-usage leaderboard, under threat of layoff, is a concrete field instance of the mechanism the DORA-based talk describes: top-down AI utilization mandates decreasing psychological safety and producing metric gaming rather than genuine productivity gains.
explicitClear AI policies have the strongest positive association with successful AI adoption (90% confidence from DORA Bayesian analysis)conf: high
explicitdevelopedAI may never reach capability levels sufficient to replace human engineers, and the improvement trajectory is flatteningconf: medium
refines ←Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard · p13 — Bernhard narrows the replacement question: agents already fully replace developers on well-bounded tasks, even if wholesale engineer replacement may never come.
explicitcorroboratedCode generation is not the bottleneck in most engineering organizations; interruptions, context switching, and meetings consume far more timeconf: high
refinesElizabeth Ayer on Mastodon — "The fact that we are *not* seeing wildly improving software all around us tells us everything we need to know" · p4 — Agrees code generation is not the bottleneck and names the actual time sinks: interruptions, context switching, and meetings.
supports ←The Office Amenity That Actually Improves Teamwork · p5 — Both hold that interruptions, context-switching, and meetings, not the visible work itself, consume the scarce resource. Corroborates the 'meetings and interruptions are the real time-sink' claim from the knowledge-worker side.
explicit⚡ disputed & corroboratedOrganizations that successfully adopt AI should hire more engineers, not fewer, because per-engineer throughput increasesconf: medium
contradicts ←The Agentic P&L: Beyond the Empire of Headcount · p7 — Forecasts labor contracting sharply (400 to 80-100 residual humans), incompatible with the claim that successful AI adopters should hire more engineers, not fewer.
supports ←The AI paradox: More automation, more humans, more work | Dan Shipper · p4 — Every's doubled headcount at an AI-forward company is a live case of the claim that successful AI adopters should hire more engineers, not fewer.
explicitcorroboratedPsychological safety is the most important characteristic of high-performing engineering teams, and this applies especially now during AI uncertaintyconf: high
supports ←Guide to scaling engineering organizations · p10 — Both independently anchor on psychological safety (via Project Aristotle) as the single most important determinant of team effectiveness -- this piece documents the pre-AI mechanism (explicit written culture/communication guides) for deliberately building it, which the later AI-era piece argues is now under additional pressure from top-down AI mandates.
explicitdevelopedJunior engineers adopt AI faster but senior engineers ultimately save more time from itconf: medium
refines9) Nils Brede Moe om AI-paradokset, autonomi, læringskultur og fremtidens team · p9 — Adds an adoption-speed dimension to the senior/junior asymmetry: juniors adopt AI faster, but seniors ultimately extract more value from it.
refines ←Engineering Leadership Report 2026 · p12 — Postman's report treats 'AI harms junior developers' as a near-consensus belief (84%) without further differentiation. The DORA-based talk offers a more granular data point — junior engineers adopt AI faster, but senior engineers ultimately save more time from it — that complicates the blanket 'AI is bad for juniors' framing rather than simply confirming or denying it.
explicitIf you are not using metrics to drive improvement, collecting them is pointlessconf: high
implicitThe ROI of AI adoption is primarily determined by organizational leadership practices, not by the technical capability of the AI tools themselvesconf: high
explicitEngineering productivity was poorly defined and measured even before AI, and we are now compounding that confusionconf: high
explicitSystem prompts / agent markdown are a governance lever that requires organizational feedback loops, not just initial configurationconf: high

Arguments

Leaders should frame AI as throughput augmentation, not cost reduction strength: high

Explicit premises:

Per-company data shows enormous variance — the same tools produce wildly different outcomes
Organizations with clear policies, learning time, and psychological safety see positive results
Theory of Constraints says throughput, not cost, is the job of the machine of business
Zapier is hiring more because per-engineer ROI increases

Implicit premises:

Engineers who fear replacement will not adopt AI effectively, creating a self-fulfilling prophecy of poor ROI
The labor market for engineers remains competitive enough that fear-driven attrition is a real cost
Throughput gains from AI are large enough to justify increased hiring budgets

Inferences:

Organizations that frame AI as cost-cutting will lose their best engineers and see negative AI ROI, creating a death spiral
The 'throughput vs. cost' framing is itself a signal of organizational health independent of AI

Single-metric AI mandates (e.g., 100% utilization) are counterproductive strength: high

Explicit premises:

Goodhart's Law: when a measure becomes a target, it ceases to be a good measure
The Cobra Effect demonstrates incentivization backfiring catastrophically
Engineers will game utilization metrics (e.g., updating README files)
SPACE framework established that productivity requires oppositional multi-dimensional measurement

Implicit premises:

Leaders who mandate single metrics are typically motivated by appearing decisive rather than by genuine understanding of AI impact
The political incentive to show AI adoption often overrides the analytical incentive to measure impact honestly

Inferences:

Any AI KPI dashboard with only utilization metrics is actively misleading leadership

AI should be applied across the SDLC, not just to code generation strength: high

Explicit premises:

Engineers get only 5-6 hours/week of coding time in scaled organizations
AI saves ~3.4 hours/week but interruptions, meetings, and context switching consume far more
Theory of Constraints: an hour saved on a non-bottleneck is worthless
Morgan Stanley saves 300,000 hours/year on reverse engineering, not code generation
Spotify's incident management agent handles 90% of incidents

Implicit premises:

The industry's fixation on code generation is partly driven by vendor marketing, since code-gen is the most visible and demo-friendly AI capability
Identifying the actual bottleneck requires the kind of measurement infrastructure most organizations lack

Inferences:

Organizations that only deploy AI for code completion are leaving most of the potential value on the table

AI will not replace human engineers in the foreseeable future strength: medium

Explicit premises:

SWE-bench top performer at 44% autonomous task completion
Improvement trajectory flattening (34% → 44% over 18 months)
95% of AI pilots fail (MIT NANDA study)
Underlying algorithms haven't fundamentally changed since the 1950s

Implicit premises:

Benchmark performance on isolated tasks is a reasonable proxy for real-world engineering capability
The current rate of improvement will not suddenly accelerate due to architectural breakthroughs

Inferences:

Leaders using AI replacement as a cost-cutting strategy are making plans based on capabilities that don't exist and may never exist

Measurement of AI impact requires a mixed-method, multi-dimensional, longitudinal approach strength: high

Explicit premises:

Aggregate averages hide per-company variance
Qualitative and quantitative metrics tell different stories (the doctor metaphor)
Survey data is unreliable without 90%+ participation, good engineering, and Goodhart-awareness
Metrics without improvement intent are pointless

Implicit premises:

Most organizations lack the analytical sophistication to implement this kind of measurement
DX (the speaker's company) is well-positioned to sell this capability — this is partly a sales argument

Inferences:

The industry will remain confused about AI ROI for years because few organizations will invest in rigorous measurement

Narratives

The Cobra Effect: an emperor incentivized cobra kills, people farmed cobras instead, and when the program was cancelled, released cobras made the problem worse
→ Supports the argument against single-metric AI mandates and Goodhart's Law — incentivizing utilization metrics will produce gaming that makes the underlying problem worse
Google's Project Aristotle found that psychological safety, not talent or resources, was the top predictor of team performance
→ Supports the proposition that psychological safety matters most and that leaders must proactively address AI displacement fears
Zapier has more bots than humans, reduced standups from 5x to 2x/week, cut onboarding from 30 minutes to 2 weeks, and is hiring more than ever
→ Supports the throughput-over-cost framing — the most AI-forward company is hiring more, not fewer, engineers
Morgan Stanley's DevGen.AI agent reverse-engineers legacy COBOL/Perl specs, saving 300,000 hours/year
→ Supports applying AI to non-code-generation bottlenecks in the SDLC
Spotify 2.0: SRE incident agent provides context and runbook data in Slack within the first critical minutes, handling 90% of incidents
→ Supports SDLC-wide integration — AI applied to incident response rather than code generation
The METR study showed 19% productivity decrease despite engineers feeling more productive
→ Supports the proposition that perception and reality diverge and that measurement is essential — you cannot trust subjective assessment of AI productivity
The doctor analogy: 'I don't feel well' vs. 'your vitals are fine' — you need both qualitative and quantitative data
→ Supports mixed-method measurement — system metrics without developer context are insufficient

Questions addressed

⭕ What is the actual ROI of AI in engineering organizations?
Modest positive on average, but enormous per-company variance. The answer depends on leadership practices, not the tools.
✅ How should leaders measure AI impact?
The DX AI Measurement Framework: oppositional metrics across utilization, impact, and cost dimensions, using mixed methods (telemetry, experience sampling, surveys) longitudinally
✅ Will AI replace engineers?
No, not now and possibly never. SWE-bench at 44%, trajectory flattening, 95% pilot failure rate. Frame as augmentation.
✅ How should leaders address engineer fear of AI displacement?
Proactive transparency, frame as throughput not cost-cutting, share data on AI limitations, invest in learning
✅ Where should AI be applied beyond code generation?
Across the SDLC: incident management, code review, PRD generation, legacy reverse engineering, onboarding
⭕ What organizational factors differentiate AI winners from losers?
Clear policies, learning time, psychological safety are identified, but the causal mechanism is not deeply explored — correlation from DORA data

Gaps

incompleteThe presentation treats 'leadership practices' as the causal variable explaining per-company variance, but does not control for confounders like codebase quality, technical debt, team size, or domain complexity. Companies that already have good DevEx practices may simply be better at everything, including AI adoption.
Read next: Research on organizational readiness factors for technology adoption; look at DX's own AI Impact Report for more controlled analysis
incompleteThe argument against AI replacing engineers relies heavily on current SWE-bench scores and a claim about algorithms not changing since the 1950s. This understates the pace of agentic and scaffolding innovations (MCP, RAG, multi-agent systems) that improve effective capability without changing underlying model architecture.
Read next: Anthropic, OpenAI, and DeepMind research on agent scaffolding; François Chollet's critique of benchmark-as-capability proxies
incompleteNo discussion of how AI changes the nature of engineering work itself — what skills become more/less valuable, how code review practices should adapt to AI-generated code, or whether existing DevEx frameworks designed for human-only teams remain valid when agents are team members.
Read next: Kent Beck's work on AI-assisted development; Addy Osmani's Agent Skills framework; research on human-AI teaming from Microsoft Research
closed byThe Orchestration Tax is You — Answers how AI changes the nature of engineering work: architecture/system-design around the attention bottleneck becomes the urgent skill, and review practices must adapt via batching, offloading machine-verifiable checks, and fleet sizing (p9, p11, p12, p19).
incompleteDX is both the research organization and the vendor selling the measurement platform. The talk acknowledges DX's role but does not address how this commercial interest might shape which metrics are recommended or how 'success' is defined.
Read next: Independent developer productivity research (e.g., DORA team at Google, academic DevEx studies not commercially affiliated)
uninformedThe claim that 'underlying algorithms haven't changed since the 1950s' is misleading. While perceptrons date to the 1950s, transformers (2017), RLHF, constitutional AI, chain-of-thought, and scaling laws represent genuine algorithmic innovations. The framing conflates 'neural networks exist since the 1950s' with 'nothing has changed.'
Read next: Vaswani et al. 'Attention Is All You Need' (2017); Kaplan et al. scaling laws papers; survey of post-transformer architectural innovations

Presentation: Leadership in AI-Assisted Engineering

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Related documents