Measuring AI Impact

Distinguishing real AI productivity effects from adoption theater — longitudinal, oppositional and outcome-anchored measurement of what AI actually changes.

Also known as: amplification, augmentation, human-equivalent hours, time to 10th PR, Time to 10th PR, true throughput, oppositional metrics, first-pass acceptance rate, First-pass acceptance rate, metrics that matter (override rate, impact on P&L), AI ROI, daily AI use, ramp-up time, throughput, AI-Generated Content Usage (AIGCU), AIGCU

How authors use it

Oldest first (extraction date when no publication date is known).

Documents (55)

• Closing the Gap Between AI Investment and Financial Return David L. Rogers, Krishnan Sankaranarayanan · 2026-09-23 Reader 🖥️
  Because enterprise AI value is the product of three multiplicative factors—technology, adoption, and strategic alignment—rather than their sum, companies fail to convert heavy AI investment into measurable financial return primarily because they neglect the most overlooked and highest-leverage factor, strategic alignment, and must instead anchor every AI investment to their own value drivers, industry-value-chain position, and income-statement economics rather than pursuing generic, vendor-shared use cases.
• Measuring AI's Unintended Consequences James Shore · 2026-09-22 Reader 🖥️
  Because AI's unintended consequences — declining maintainability, burnout, and vendor/token lock-in — carry real economic costs but, unlike delivery speed, cannot be measured by a controlled trial and instead surface only gradually and noisily, organizations should track a cluster of imperfect lagging indicators (chiefly value-add percentage, backed by DORA metrics and token usage) and build explicit long-horizon scenario models (a maintenance-cost 'tax,' an excess-code-churn model, and a lock-in variant of both) to inform — never replace — judgment, even though the most trusted of these indicators is itself unusually easy to game.
• Measuring AI's Impact on Delivery Speed James Shore · 2026-09-15 Reader 🖥️
  Because no unit of software output is standardized and raw on-time-delivery rates are corrupted by chronic estimate over-optimism, an organization can only find out whether AI is really making delivery faster by running a randomized, within-subject, repeated-measures trial that compares each person's or team's own 'with AI' and 'without AI' actual-to-estimate ratios, collected before assignment is known, under an otherwise identical process.
• Stop Automating Old Processes. Design New Ones Instead. Masha Shunko and Serguei Netessine · 2026-09-14 source ↗ Reader 🖥️
  Enterprise AI investment fails to produce enterprise-wide returns not because companies under-automate but because they redesign individual tasks instead of the surrounding workflow, so managers must treat the workflow — not the task — as the unit of redesign, deciding explicitly what to change, where human authority remains, and how the process degrades when the model is uncertain or wrong.
• Are You Getting Your Money's Worth from AI? James Shore · 2026-09-08 Reader 🖥️
  Because AI's costs and benefits in software development are real, substantial, and easily misjudged by anecdote, self-assessment, or gut feel, organizations facing consequential tradeoffs between speed, cost, and people's jobs must rigorously measure both sides — using durable, outcome-anchored, approach-level metrics rather than gameable activity proxies like lines of code, pull-request count, or individual performance targets.
• Living and Working in the Age of Longevity HBR.org · 2026-08-19 source ↗ Reader 🖥️
  Because people are living and staying healthy much longer, the traditional three-stage life (educate, work, retire at 60-65) and its career-ladder metaphor no longer fit, so individuals must take up deliberate personal agency to 'weave' a life across eight threads of productivity and nurture — while organizations and institutions, which still lag behind this shift, need to stop using age as a proxy for capability and instead build the trust, performance-management, and flexibility that let people work productively, and humanly, well into their 70s, 80s and beyond.
• Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents Leela Kumili · 2026-08-17 source ↗ Reader 🖥️
  Grab reduced the mechanical share of its analysts' work by deploying AI agents across a staged autonomy model, an agent-routing system, and continuously maintained certified data context, while deliberately keeping humans accountable for metric definitions, causal interpretation, and final decisions.
• TBM 437: Tokens, Hours, Points, and Other Curious Proxies John Cutler · 2026-08-17 Reader 🖥️
  Tokens are the latest in a lineage of seductive measurement proxies -- hours, story points, flow metrics -- that organizations mistake for a theory of value because they happen to be unusually easy to measure, and the fix is not a better proxy but making the underlying causal model of value explicit (treating the team as the cost unit, funding durable lanes, stating falsifiable hypotheses) so that ROI becomes the *output* of that model rather than a number computed directly from an easy-to-meter input.
• How Organizations Use AI: Evidence From ChatGPT Aaron Chatterji, David Holtz, Neel Rakholia, Prasanna Tambe, Gawesha Weeratunga · 2026-08-11 Reader 🖥️
  Enterprise ChatGPT adoption and use, measured through OpenAI's own product telemetry across 1,500+ organizations and 17M+ messages, is a broad but highly uneven organizational phenomenon — concentrated among larger, more valuable, more intangible-capital-intensive firms, and diffusing unevenly across job functions, seniority levels and tasks within adopting firms — showing that organizations are still in an early, open-ended process of learning where generative AI belongs in their workflows rather than having already converted access into uniform deployment.
• AI-Assisted Software Development: Team Profiles and Capabilities for Putting Research into Action Ben Linders · 2026-07-30 source ↗ Reader 🖥️
  DORA's 2025 research reframes AI adoption in software development as an organizational-system problem rather than a tooling problem: AI amplifies whatever team dynamics already exist, and the return on AI investment comes from developing identifiable, data-derived capabilities (from a clear AI stance to quality internal platforms) matched to your team's actual profile, not from adopting the technology itself.
• The Economic Benefit of Refactoring Giles Edwards-Alexander (published on martinfowler.com, byline Martin Fowler) · 2026-07-30 source ↗ Reader 🖥️
  Refactoring an agentic codebase according to a strict, correctness-preserving discipline can pay a measurable economic dividend — in this experiment, an 83% reduction in the input-token cost of an identical representative future change — because it lets an AI agent read a smaller, more targeted slice of the code to make the same change, even though the total code volume barely shrinks.
• Research: How AI Agents Broaden the Scope of Knowledge Work Jeremy Yang, Kate Zyskowski, Noah Yonack, Jerry Ma (Perplexity) · 2026-07-29 Reader 🖥️
  AI agents don't just make existing knowledge work faster — by combining much longer autonomous execution with far broader tool and context integration, they lead people to attempt fundamentally different, more cross-disciplinary, and previously-impractical work, so leaders must redesign roles, review points and team boundaries around this expanded scope rather than just harvesting the time and cost savings.
• Uber's Zero Growth Stack: Scaling Services, While Optimising Infrastructure and AI Cost Olimpiu Pop · 2026-07-28 source ↗ Reader 🖥️
  Uber's 'Zero Growth Stack' shows that sustaining service growth without proportional infrastructure growth now requires treating both compute resources (via dynamic, cgroup-aware GC tuning like GOGCTunner) and AI-tool spend (via usage caps and proposed code-quality-linked metrics) as continuously governed control loops rather than static, one-time settings or unconstrained rollouts.
• Google's AlphaEvolve Reaches General Availability with Evolutionary Code Optimization as a Service Steef-Jan Wiggers · 2026-07-19 Reader 🖥️
  AlphaEvolve's move from DeepMind research project to generally-available Google Cloud product proves that LLM-driven evolutionary code optimization delivers real, substantial engineering gains, but only inside the narrow band of problems that already have (or can be given) a precise, automatable evaluation function -- outside that band, in the 'messy' business logic that makes up most production software, the approach currently has no demonstrated purchase, and the announcement's vendor-only evidence means even the in-band gains should be read as directional, not verified.
• TBM 431: The Denominator That Matters John Cutler · 2026-07-19 Reader 🖥️
  AI success in an organization is the multiplicative product of Machine Understanding, Problem Understanding, and Practice Evolution, all divided by the Social Contract -- and because the Social Contract sits in the denominator of a multiplicative formula rather than as a fourth additive factor, no amount of strength in the numerator can compensate for a broken Social Contract, which is why resistance to AI mandates absent a credible social contract is rational self-preservation rather than ignorance or fear.
• AI Enthusiasts Are in a Race Against Time, AI Skeptics Are in a Race Against Entropy Charity Majors · 2026-07-11 Reader 🖥️
  The AI enthusiast-versus-skeptic split inside high-performing engineering orgs is not one side being right and the other wrong but two groups each responding to a real existential threat (missed windows vs. systems decaying into slop) that they never see the other's side of, and the cure is to rebuild a shared reality by telling the whole story and then treat AI adoption as an engineering problem rather than a rhetorical war.
• Five studies that are changing how I think about AI in software engineering Engineering Enablement · 2026-07-10 Reader 🖥️
  Five methodologically different studies converge on one story: AI has compressed the upstream code-generation work of software engineering, shifting the binding constraint downstream to review, integration, verification, and shared understanding, while our tools, metrics, and team designs have not yet moved with it.
• The Pulse: Interesting AI coding stats from Cursor The Pragmatic Engineer (Gergely Orosz) · 2026-07-09 Reader 🖥️
  Cursor's two years of aggregated usage data show that AI-coding output is extremely power-law skewed across developers, that its cost is dominated by input/context-reading tokens (making caching decisive), and that developers are rapidly abandoning manual review of AI-written commits.
• Code maintainability plummets in the AI coding era Bill Doerrfeld · 2026-07-07 source ↗ Reader 🖥️
  New longitudinal telemetry (GitClear/GitKraken, 623M commits, 2023-2026) shows that since the AI-coding era began, software maintainability has degraded across eight code-quality metrics -- duplication and error-masking are up while code reuse and legacy refactoring have collapsed -- trading short-term generation speed for compounding technical debt.
• Agentic Code Review Addy Osmani (addyosmani.com) · 2026-07-06 source ↗ Reader 🖥️
  Because agents made writing code nearly free while understanding it stayed exactly as costly as ever, code review has become the binding constraint on shipping software — and since review must now reconstruct an intent that never existed in any human mind, the right amount of human review is not a fixed rule but a dial set by each change's blast radius.
• AI productivity is burning out your best engineers Sreenivasa Reddy · 2026-07-06 source ↗ Reader 🖥️
  AI coding tools quietly transfer the uncredited work of validating and fixing AI output onto mid-level engineers -- the 'invisible validator' burden -- burning out the exact cohort an organization can least afford to lose, and the remedy is structural rather than personal: leaders must make oversight explicit, measured, and rotated so the hidden cost stops falling silently on the middle.
• SWLW #710: Revised rules of Eng Leadership, When AI costs more than the engineer, and more. Oren Ellenbogen · 2026-07-06 Reader 🖥️
  As AI reshapes engineering, the decisive leadership work shifts to human judgment applied to fundamentals — reaffirming sound operating principles, justifying AI's now salary-scale cost, and demanding engineering rigor — because AI only amplifies (in magnitude, not direction) whatever practices, clarity, and dysfunction already exist.
• Please stop the AI Confidence Theater Elena Verna (guest essay, cross-posted by Lenny Rachitsky / Lenny's Newsletter) · 2026-07-02 source ↗ Reader 🖥️
  The widespread performance of confident AI mastery — inflating modest, unreliable AI wins into 'life-changing' claims and projecting certainty where genuine inconsistency exists — does more harm than good, and everyone up and down the org chart should replace it with honest anchoring in real, verifiable outcomes.
• Should You Still Become a Software Engineer in 2026? GitHub VP Jean Lee (interviewer); Scott Hanselman (guest, VP & Member of Technical Staff, Microsoft/GitHub) · 2026-06-25 source ↗ Reader 🖥️
  AI coding agents are the latest in a recurring cycle of developer 'power tools' that change the tools but not the craft or its human accountability, so the real risk of the AI era is not that engineers become obsolete but that the profession fails to deliberately re-engineer its junior-to-senior training pipeline (via something like nursing's preceptor model) before AI hollows out the tasks juniors traditionally learned on.
• You can't unit test for taste Karl Tryggvason · 2026-06-22 source ↗ Reader 🖥️
  In software you can assert correctness with red/green tests, but taste — judging what output is good, relevant and well-proportioned where there is no ground truth — cannot be unit-tested, so it stays an irreducibly human, iterative judgment even when AI does the production work.
• AI coding creates two kinds of debt. You're only measuring one Antonija Bilić Arar · 2026-06-18 source ↗ Reader 🖥️
  AI-generated code creates a second, largely invisible kind of debt — cognitive debt, the erosion of shared team understanding of what a system does and why, distinct from ordinary technical debt in the code itself — and it is a genuine measurement blind spot: satisfaction scores, NPS and mentor ratings can hold steady or rise while it accumulates, so organizations must replace satisfaction-based signals with trajectory-based skill measurement and deliberate team-level questions to catch it before it becomes irreversible.
• The 8 software engineering metrics AI broke Michael Hill · 2026-06-16 source ↗ Reader 🖥️
  Software engineering metrics were built on the assumption that human effort and output move in rough proportion, and AI has severed that link by making activity nearly free — so activity/volume proxies (deployment frequency, cycle time, PR and commit volume, lines of code, satisfaction, efficiency/flow, DevEx/DXI) now measure tool adoption rather than engineering quality, while only outcome-based measures (time to recover, escaped defect rate, delivered value) remain trustworthy, because they measure what happened rather than how much happened.
• AI isn't making developers more productive – it's making them busier Chris Stokel-Walker · 2026-06-11 source ↗ Reader 🖥️
  A large-scale study of 100,000+ GitHub developers reframes the AI-coding story as redistribution rather than acceleration: AI makes developers busier by ballooning code output, but net shipped value barely moves because fixed human review/integration capacity is the binding downstream constraint.
• Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard Fabrice Bernhard · 2026-06-06 Reader 🖥️
  The headline 17.3x speedup that AI async agents produce at the start of the software value stream collapses to a 1.3x increase in release frequency at the end of the stream — because human and organisational bottlenecks (unclear business needs, organisational dependencies) dominate end-to-end throughput; therefore AI creates real value only when it is targeted at the actual bottleneck, and identifying that bottleneck is what Lean Tech is for.
• Engineering Leadership Report 2026 Postman · 2026-06-05 Reader 🖥️
  Engineering leadership is undergoing genuine, compounding transformation — organizational flattening (now reaching upper management), expanding technical-plus-managerial scope, and AI adoption that has outpaced governance and measurement — and the organizations that come out ahead in 2026 will not be the ones that adopted AI fastest but the ones that built the trust, clarity, and investment in people that let AI amplify rather than erode human capability.
• How People Are Really Using AI in 2026 Harvard Business Review · 2026-06-01 Reader 🖥️
  Across a third annual edition, generative-AI use is widening into an ever-broader and more intimate range of tasks — best read as shifts in emphasis, not ruptures — with people increasingly outsourcing both their thinking ('thinkslop') and their emotional lives to AI, even as, in business, the technology so far yields mostly marginal efficiencies rather than game-changing transformation, leaving the survival of human agency as the open question.
• Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools Mert Demirer, Leon Musolff, Liyuan Yang · 2026-05-27 source ↗ Reader 🖥️
  Across three successive generations of AI coding tools, large task-level productivity gains (up to 180% more commits) attenuate sharply as they propagate up the software production hierarchy to final shipped output (only 20-30% more releases), because AI and downstream human effort are strong complements rather than substitutes (estimated elasticity of substitution of about 0.25) — so increasing automation of code-writing alone yields only bounded gains in what actually reaches users, a pattern that also shows up in aggregate GitHub activity and across four app marketplaces, where new app releases have risen but end-user usage has not.
• Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) J. Paul Reed · 2026-05-22 Reader 🖥️
  The ironies Bainbridge identified in 1983 — that automation makes the human operator more, not less, crucial while simultaneously eroding the skills needed to intervene — persist and intensify under AI, so engineering organizations must treat AI as an efficiency lever that demands deliberate cultivation of human thoroughness, mental models, and coordination practices rather than as a substitute for them.
• Itchy Brain (Still Burning: Michael Grinich & Kent Beck on AI adoption, enterprise developer tools) Kent Beck — Still Burning Podcast · 2026-05-20 Reader 🖥️
  We are not in another technology hype wave but in a step-function transformation: the marginal cost of producing software is collapsing toward zero, large incumbents are adopting AI faster than any prior shift, and the whole ecosystem — not just the AI labs — is accelerating; in this 'Red Queen' regime, profit and productivity may not visibly improve because every competitor is running at the new tempo, so the only sane response is to paddle straight into the wave by sustaining the maker's compulsion (the 'itchy brain'), reshaping engineering leadership toward hands-on impact, and accepting that software engineering as a profession will resemble medicine or law more than the unique 2010s template that minted six-figure undergrads.
• Reinventing team flow within AI uncertainty Avanscoperta Blog · 2026-05-14 source ↗ Reader 🖥️
  Because large language models can touch nearly every step of software delivery at once rather than one isolated area, teams should respond to AI-driven uncertainty not with individual-level incentives or isolated experiments but with continuous, team-owned experimentation — resetting rituals, replacing estimation with throughput, and shifting from fixed roles to dynamic contributions — because the alternative (local, uncoordinated optimization) can make the whole system worse even when each individual change looks like a win.
• The model doesn't matter anymore – and other AI skills worth your Thursday Turing Post · 2026-05-05 source ↗ Reader 🖥️
  Professionals need to identify which AI skills will remain valuable in 2026, and a free virtual AI Skills Conf on May 14 promises practical, practitioner-led sessions on AI ROI, agentic memory, AI workflows, and professional irreplaceability.
• Presentation: Leadership in AI-Assisted Engineering Justin Reock · 2026-04-20 Reader 🖥️
  Engineering leaders who frame AI as a throughput multiplier rather than a cost-cutting tool — and back that framing with clear policies, learning time, psychological safety, oppositional measurement, and SDLC-wide integration — are the ones whose organizations see positive ROI from AI adoption, while top-down utilization mandates reliably backfire.
• SWLW #702: How I created OpenClaw, Your CEO is suffering from AI Psychosis, and more. Oren Ellenbogen · 2026-04-20 Reader 🖥️
  Effective tech leadership requires anchoring in measurable value and compounding effort rather than chasing the appearance of productivity, whether through AI theater, performative suffering, or shallow busyness.
• Technology Radar, Volume 34 Thoughtworks (Technology Advisory Board) · 2026-04-14 source ↗ Reader 🖥️
  AI-assisted, agentic software development is moving faster than the industry's ability to evaluate it, which means teams must simultaneously retain long-held craftsmanship principles (testability, deliberate design, feedback loops) and rethink team/agent topologies, trust boundaries and metrics that were built for a pre-agent world.
• Developer ramp-up time continues to accelerate with AI Engineering Enablement · 2026-04-09 Reader 🖥️
  AI-assisted developers are reaching their 10th merged PR in roughly half the time compared to pre-AI baselines, with the trend continuing to accelerate quarter-over-quarter, though this metric captures speed-to-milestone without measuring the quality or depth of understanding behind those contributions.
• The Acceleration Whiplash Faros · 2026-04-09 Reader 🖥️
  AI coding tools have crossed, without any deliberate organizational decision, from assistant to primary author of code, and two years of telemetry from 22,000 developers show that this shift is producing a widening gap between throughput (up) and production quality (down) that holds regardless of engineering maturity -- so organizations must raise code quality at the point of AI authoring rather than add more downstream review.
• Measuring AI impact, assessing readiness, and new data trends DX · 2026-04-03 source ↗ Reader 🖥️
  Engineering leaders should move beyond naive AI adoption metrics toward longitudinal measurement that distinguishes amplification (making humans more productive) from augmentation (agents as additional capacity), while recognizing that the real bottlenecks to AI ROI are not code generation but developer experience, human judgment, and organizational readiness -- the same problems that preceded AI.
• Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work Brandon Lepine; Juho Kim; Pamela Mishkin; Matthew Beane · 2026-03-10 Reader 🖥️
  Across 34 finance professionals performing a complex 10-hour valuation task with GPT-4o, AI-generated content usage is positively associated with quality but is dominated by an independent, additive, and roughly three-times-larger negative association with extraneous cognitive load; extraneous load is not a static property of any single response but a path-dependent, within-speaker-persistent feature of the interaction jointly constructed by user and model — and the design response is 'precision proactivity', i.e. dialogue systems that detect entrenched high-load trajectories, constrain system-initiated scope expansion (especially unsolicited task switching), and structure disclosure so users can regulate intake without absorbing imposed reorganization costs.
• Patterns for Reducing Friction in AI-Assisted Development Rahul Garg · 2026-02-24 source ↗ Reader 🖥️
  AI coding assistants produce misaligned output not because they lack capability but because developers skip the same onboarding, design discussion, and standards-sharing rituals that make human pair programming effective, and adopting five collaboration patterns that mirror those rituals can transform AI from a source of friction into a force multiplier.
• AI doesn't create great developers, it amplifies them Bill Doerrfeld · 2026-01-20 source ↗ Reader 🖥️
  The apparent productivity gap between heavy and light AI users is mostly a selection effect, not a treatment effect of AI itself — the developers getting the most out of AI coding tools were already the most productive developers the year before, so AI amplifies pre-existing talent rather than creating new talent, and leaders should stop using general AI adoption as a performance proxy.
• Thomas Nys' Post Thomas Nys · 2025-12 Reader 🖥️
  Organizations can't get AI initiatives to scale because a large, usually invisible share of engineering capacity (the author cites 20-40% industry estimates) is already consumed maintaining accumulated technical-debt shortcuts, so AI pilots stall not for lack of model quality or team size but because they're being layered on fragile, duct-taped foundations that were never fixed first.
• Simon Wardley's Post — Industrialisation of ML: nine notes on what to expect Simon Wardley · 2025-12 Reader 🖥️
  What is happening with AI is the industrialisation of machine learning — a one-way evolutionary shift to commodity/utility — and the right posture toward it is neither resistance nor hype but disciplined adaptation through the predictable 5-8 year co-evolution of practices, with explicit attention to where bottlenecks, supply-chain power, and inequality will actually land.
• Agentic AI in the Software Development Lifecycle (SDLC) Julie Isdahl · 2025-11-19 Reader 🖥️
  Agentic AI integrated across the software development lifecycle is a strategic transformation Telenor's technology leadership should actively sponsor and scale now, not a tooling upgrade engineering teams can adopt on their own, provided it is paired with governance, security oversight, and workforce reskilling.
• How to measure AI developer productivity in 2025 | Nicole Forsgren Lenny's Podcast (Lenny Rachitsky interviewing Nicole Forsgren) · 2025-10-19 source ↗ Reader 🖥️
  Because lines-of-code-style productivity metrics are trivially gamed by verbose LLM output and the DORA/SPACE frameworks were not designed for AI's new failure modes (earlier and faster feedback loops, nondeterministic output requiring trust verification, review time displacing writing time), measuring whether AI is actually helping developers requires starting with qualitative listening and short forced-choice surveys, adding trust as an explicit new dimension, and framing whatever is measured in the specific business terms (market share, margin, velocity) that an organization's leadership already uses -- while openly acknowledging that AI's contribution usually can't be cleanly separated from concurrent developer-experience improvements.
• The productivity paradox of AI coding assistants Lisa Dziuba · 2025-09-12 source ↗ Reader 🖥️
  AI coding assistants create a productivity placebo — they feel fast because of instant dopamine-reward feedback — but measured across independent studies they deliver marginal or negative net productivity for experienced developers on real codebases, degrade code quality and security, and the widely repeated '10x productivity' claim does not survive controlled measurement.
• Leading your engineers towards an AI-assisted future Pete Hodgson · 2025-06-26 source ↗ Reader 🖥️
  An engineering leader should drive AI-coding adoption the same way they'd drive any major capability shift — by setting a clear, phased goal (Experimentation, then Adoption, then Impact) measured with lightweight KPIs, giving teams real autonomy over how they get there, and backing that autonomy with concrete organizational support (training, budget, protected time, a community of practice, empathetic communication) — rather than mandating a usage percentage from day one.
• Time Warp: The Gap Between Developers' Ideal vs Actual Workweeks in an AI-Driven Era Sukrit Kumar, Drishti Goel, Thomas Zimmermann, Brian Houck, B. Ashok, Chetan Bansal · 2025-02-21 source ↗ Reader 🖥️
  A developer's actual workweek diverges substantially from their self-reported ideal workweek — overloaded on communication, security/compliance, and task management and underloaded on coding, architecture, and learning — and the size of that gap tracks measurably with lower productivity and satisfaction, so AI automation effort should be redirected from code-writing (already well-served) toward the specific non-coding tasks developers rate as both high-effort and low-satisfaction.
• Is Agile Dead? How AI and Extreme Agile Redefine Work Jeff Sutherland · 2024-12-02 source ↗ Reader 🖥️
  Agile is not dead, but it survives only by fusing with AI into 'Extreme Agile' — a stack of neuroscience, Scrum, and AI layers that delivers 30-100x productivity gains and makes AI-naive Scrum teams unemployable by 2030, so organizations and workers must adopt AI-augmented Agile now or face obsolescence.
• 2024_final_dora_report DORA · 2024-10-25 Reader 🖥️
  Technology investments — generative AI adoption and internal developer platforms alike — reliably raise individual, team, and organizational performance only when paired with the underlying sociotechnical fundamentals (small-batch delivery discipline, developer independence, transformational leadership, a genuine user-centric and documentation culture, and stable priorities); absent those fundamentals, the same investments degrade software delivery throughput and stability and can increase burnout, so AI and platforms amplify whatever organizational system already exists rather than fixing it.
• CannotMeasureProductivity Martin Fowler · 2003-08-29 Reader 🖥️
  Software productivity cannot be measured because its true output — delivered business value — cannot be measured, so every popular proxy (lines of code, function points, feature counts, project 'success' rates) either measures the wrong thing or measures it too late to be useful.

Propositions touching this concept (32)

By publication month, oldest first (extraction month when no publication date is known).

2024-10

• explicit⚡ disputed & corroboratedDespite those process-level and individual-level gains, a 25% increase in AI adoption is associated with a 1.5% decrease in software delivery throughput and, more sharply, a 7.2% decrease in software delivery stability.
  2024_final_dora_report · p2
• explicitInternal developer platforms raise individual productivity (+8%), team performance (+10%), and organizational performance (+6%), but reduce throughput (-8%) and change stability (-14%) relative to not using a platform, and platform-driven instability combined with the platform itself is specifically linked to higher burnout.
  2024_final_dora_report · p5
• explicitRequiring teams to use a platform exclusively for the entire application lifecycle (removing the option to break out of it) costs throughput (-6%) — mandated, inflexible platform adoption undercuts the independence that otherwise drives the platform's benefits.
  2024_final_dora_report · p7

2026-02

• explicitcorroboratedFirst-pass acceptance rate is a leading indicator for change failure rate (DORA)
  Patterns for Reducing Friction in AI-Assisted Development · p6

2026-03

• empirical findingAI-generated content usage (AIGCU) is positively associated with output quality: β≈0.27 to 0.35, p<0.01 across mixed-effects specifications (N=1,178 obs, N=33 participants).
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work · p1
• empirical findingAIGCU×EL and AIGCU×ICL interactions are consistently null — no evidence that AI usage moderates or buffers the marginal load-quality relationship. Pattern is additive, not multiplicative.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work · p4
• empirical finding (moderation)Less experienced professionals derive larger marginal quality gains per unit of AIGCU (b-path β=0.618 at −1SD experience vs 0.192 at +1SD; Experience×AIGCU β=−0.159, p<0.01).
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work · p6
• empirical finding (moderated mediation)More experienced professionals increase AIGCU uptake more sharply as extraneous load rises (a-path β=0.896 at +1SD vs 0.308 at −1SD; moderation β=0.294, p<0.001). The dissociation: those who'd benefit most don't uptake more under load.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work · p7
• empirical finding (mediation)Mediation pathway through AIGCU attenuates ~37% of intrinsic-load penalty (0.060/0.164) and ~29% of extraneous-load penalty (0.222/0.762) — partial but incomplete compensation.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work · p8
• empirical findingContent usage exhibits direct persistence across consecutive responses (β=0.115, p=0.004) — AIGCU follows its own temporal trajectory independently of load dynamics.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work · p14
• methodological claim (robustness)Direction and significance of EL→quality, ICL→quality, and AIGCU→quality associations are robust across 86 parameter combinations in a specification curve analysis spanning subtask-memory dynamics, anchoring thresholds, coherence weights, and attribution hyperparameters.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work · p23
• implication / reframeFindings refine prevailing narratives about AI augmentation: AI assistance operates ALONGSIDE — not in interaction with — the cognitive conditions of the conversation. The determining factor in mixed prior results may be parallel accumulation of extraneous processing demands, not the presence of assistance per se.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work · p29

2026-04

• explicitAverage Time to 10th PR has dropped from ~70+ days in Q1 2024 to 33 days in April 2026, a reduction of more than 50%.
  Developer ramp-up time continues to accelerate with AI · p1
• explicitcorroboratedTime to 10th PR does not measure the quality of contributions, rework generated, or depth of understanding of new hires.
  Developer ramp-up time continues to accelerate with AI · p4
• explicitestablishedCross-sectional analyses correlating AI usage with higher throughput are confounded because heavy AI users were already high-throughput coders; longitudinal analysis is methodologically superior.
  Measuring AI impact, assessing readiness, and new data trends · p3
• explicitdevelopedThe ROI of AI should be measured in two distinct buckets: amplification (human productivity gains) and augmentation (agent-driven capacity extension).
  Measuring AI impact, assessing readiness, and new data trends · p4
• explicitdevelopedOverall longitudinal throughput gains from AI are approximately 10-15%, with roughly two-thirds attributable to AI itself (i.e., 8-10% net AI effect).
  Measuring AI impact, assessing readiness, and new data trends · p5
• implicitHeightened management pressure for throughput is itself a confounding variable in longitudinal analyses -- some throughput gains may be Goodhart's Law in action.
  Measuring AI impact, assessing readiness, and new data trends · p9
• explicitAI adoption drives real throughput gains at the task and feature level: task completion per developer up 33.7%, epics completed per developer up 66.2%, code-specific tasks up 210% at the team level.
  The Acceleration Whiplash · p2
• explicitcorroboratedOrganizations should not cut engineering headcount on the basis of AI throughput gains; the engineers most at risk of being cut are often exactly the ones currently absorbing the quality gap AI is creating, and premature cuts risk forced rehiring within months once quality problems compound.
  The Acceleration Whiplash · p10
• explicit⚡ disputed & corroboratedOrganizations that successfully adopt AI should hire more engineers, not fewer, because per-engineer throughput increases
  Presentation: Leadership in AI-Assisted Engineering · p7
• explicitcorroboratedMeasuring AI-assisted development success by coding throughput alone (lines of code, PR counts) in isolation negatively shapes developer behavior, producing a flood of poorly-reviewed code that slows reviews, harms delivery, and introduces security risk; first-pass acceptance rate is a more meaningful leading indicator because it exposes the hidden rework cost and connects directly to DORA's change failure rate and lead time.
  Technology Radar, Volume 34 · p10

2026-05

• explicitTraditional point estimation becomes close to worthless under AI-driven variability, so throughput is a better metric for tuning the pace of experimentation against strategic deadlines, because it protects system observability without speculative sizing effort and guards against local optimizations that clog the actual bottleneck.
  Reinventing team flow within AI uncertainty · p9
• implicitBecause the model treats augmentation and automation as mutually exclusive at a given layer for a given tool (a corner-solution result, not a direct empirical observation), any apparent blend of the two effects for a tool at a layer actually reflects different users or different sessions using the tool in different modes, not a stable intermediate mode.
  Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools · p12
• implicitBy making 'shipped and used' software the terminal measure of value, the paper's own hierarchy implicitly defines productivity around throughput-to-release rather than around other candidate outcomes (defect rates, maintainability, user-reported value) that this corpus's other sources treat as co-equal or more important consequences of the same AI-driven volume increase.
  Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools · p13
• explicitThe AI Skills Conf on May 14 offers practical sessions on AI ROI, agentic memory, AI workflows, and professional irreplaceability.
  The model doesn't matter anymore – and other AI skills worth your Thursday · E2

2026-06

• implicitCommit, line-of-code and PR counts are being mistaken for productivity: because throughput is set by the slowest (human) stage, the time AI saves upstream is reabsorbed as growing work-in-progress rather than converted into delivered value, so busyness rises while net productivity does not.
  AI isn't making developers more productive – it's making them busier · p7

2026-07

• explicitKlarna doubled its ML training throughput using AlphaEvolve, exploring roughly 6,000 candidate programs over three weeks while maintaining the bit-exact reproducibility that financial services regulation requires.
  Google's AlphaEvolve Reaches General Availability with Evolutionary Code Optimization as a Service · p5
• explicitGoogle's own internal use of AlphaEvolve predates the GA product: it has optimized silicon design for next-generation TPUs, reduced write amplification in Spanner's LSM-tree compaction by 20 percent, and cut storage footprint by 9 percent.
  Google's AlphaEvolve Reaches General Availability with Evolutionary Code Optimization as a Service · p10
• explicitcorroboratedAmplification is a magnitude, not a direction: AI amplifies whatever already exists; teams with solid fundamentals amplify in useful directions, teams without amplify the mess.
  SWLW #710: Revised rules of Eng Leadership, When AI costs more than the engineer, and more. · p5
• explicitcorroboratedAggregate cost/throughput tracking is insufficient to judge whether AI-driven engineering is worth its cost, so Uber is moving toward granular metrics — 'net code quality ratio' (AI- vs human-authored hotfix frequency) and 'compute efficiency per feature.'
  Uber's Zero Growth Stack: Scaling Services, While Optimising Infrastructure and AI Cost · p8

2026-08

• explicitManufacturing-derived flow metrics (cycle time, throughput, WIP) require 'classes of service' segmentation to remain meaningful; without it, changes in throughput or cost may simply reflect a changed mix of work rather than a real performance change.
  TBM 437: Tokens, Hours, Points, and Other Curious Proxies · p5

Tensions

Open gaps (147)

• incompleteTip 8 concedes autonomous loops ACCELERATE software entropy and prescribes guardrails (clean codebase first, explicit quality bar, commit-blocking feedback loops, plus Tip 11's jscpd Duplication Loop and Entropy Loop) as the counter — but the only quantified result offered is a coverage figure (16% to 100%), which measures throughput, not maintainability. It provides no evidence that these guardrails actually reverse the duplication / reuse-collapse / error-masking decay that longitudinal telemetry attributes to the AI-coding era, so the claim that the tips prevent maintainability debt is asserted, not demonstrated.
  11 Tips For AI Coding With Ralph Wiggum — read next: Longitudinal maintainability telemetry (duplication, reuse/'Move', legacy refactoring, obfuscation, functional connectivity) and studies that segment AI-authored from human-authored code, to test whether guardrailed loops bend those curves back.
• illogicalThe article's own framing is internally inconsistent about scope: the intro promises 'ten metrics that are breaking down,' the SPACE section is introduced as covering 'a couple of metrics' but actually covers three, and the title promises '8' broken metrics — which does match the count of metrics explicitly presented as broken (3 DORA + 3 SPACE + 2 standalone), but only by excluding the 3 counter-example metrics from the 'ten' the intro promised. The total metrics actually discussed is 11, not 10, and 'a couple' undercounts by one. This suggests the taxonomy was fitted to a round headline number rather than derived from a principled classification.
  The 8 software engineering metrics AI broke — read next: Compare against the live leaddev.com page for possible post-publication editing (the mismatch is exactly the kind an editor introduces when trimming or adding an example late); no corpus cross-reference needed.
• uninformedThe central claim that three metrics (time to recover, escaped defect rate, outcome/value measures) 'still hold up' under AI is argued purely by category (outcome vs. activity) and supported by zero data, study, or even anecdote — in sharp contrast to the detailed, named-practitioner mechanism given for each of the 8 metrics declared broken. Nearly every other specific claim in the piece is sourced to quotes from individuals at vendor-adjacent companies (Jellyfish sells engineering-intelligence software; Radixweb is a technology consultancy; Aviator sells PR/merge-automation tooling) rather than to any independent benchmark or controlled study.
  The 8 software engineering metrics AI broke — read next: This corpus's extract-ai-coding-two-kinds-debt.json documents the same general phenomenon (a comfort-metric like satisfaction/NPS rising while something real falls) with an actual RCT (N=52) and an 8-cohort/225-intern longitudinal study — read that piece's evidentiary structure as the bar this article does not meet. Look for DORA/Accelerate's own published research on whether MTTR specifically has been measured, not just asserted, as AI-invariant.
• incompleteThe vendor-interest and measurement caveats are flagged but not reconciled. Osmani notes CodeRabbit and Faros sell into this market and that some of GitClear's 12% is selection bias, yet never asks whether the effect sizes survive de-biasing, nor whether 'four independent datasets that agree' truly measure one construct — churn, incidents-to-PR, defect rate, review duration and issue counts are different metrics, and their agreement is asserted rather than shown to be about the same underlying thing.
  Agentic Code Review — read next: Independent, non-vendor telemetry on AI code quality; how churn / defect-rate / review-duration metrics are operationally defined across Faros, CodeRabbit and GitClear and whether they are comparable; selection-bias correction in AI-productivity studies.
• incompleteThe economics of the recommended stack are never priced. Running two-plus heterogeneous AI reviewers on everything, plus a human triage layer and strict deterministic gates, has a token and attention cost that is never set against the modest ~12% real productivity gain the same data reports — leaving open whether the review apparatus consumes the very productivity it is meant to protect.
  Agentic Code Review — read next: Cost/benefit accounting for multi-reviewer setups (token spend per PR, false-positive triage time); whether heterogeneous review is cost-effective only when invoked sparingly at high-signal moments rather than on everything.
• incompleteThe 'measure trajectory, not satisfaction' prescription rests entirely on Infobip's own internal, non-independently-audited alumni survey (N=96, self-selected respondents, retrospective self-assessed skill ratings) — the very instrument vulnerable to the Dunning-Kruger miscalibration the study itself measures. The article pairs this with a rigorous RCT (Anthropic, objective comprehension testing) but never flags that its own headline metric is far weaker methodologically than the study it borrows credibility from.
  AI coding creates two kinds of debt. You're only measuring one — read next: Cross-reference the Anthropic RCT's comprehension-test methodology (extract-cognitive-surrender#p4) against Infobip's self-report design; look for Storey's own published paper on triple-debt measurement (referenced elsewhere in this corpus as arXiv 2603.22106, per extract-osmani-intent-debt) to see whether it offers a non-self-report instrument.
• incompleteThe Infobip pre-AI vs. AI-era cohort comparison is a non-randomized, before/after natural comparison spanning 2022–2026 — a period during which the internship's content, mentor selection, hiring bar or applicant pool could plausibly have shifted for reasons unrelated to AI. The piece treats the causal attribution to AI as settled ('This confirmed that the problem wasn't the quality of the program') without directly ruling out these confounds.
  AI coding creates two kinds of debt. You're only measuring one — read next: DORA/Accelerate-style guidance on longitudinal measurement validity and confound control; look for a matched or randomized version of this comparison (e.g., interns with AI access experimentally withheld) that would isolate the AI effect from era effects.
• incompleteThe writeup omits the underlying study's methodology: how 'productivity' was operationalized beyond activity proxies, whether commit/PR/release counts are telemetry or self-report, how developers were assigned to (or self-selected into) the three tool generations, and what confounders (tenure, repo type, developer selection) were controlled. The headline figures cannot be assessed without the primary source.
  AI isn't making developers more productive – it's making them busier — read next: The primary source — NBER working paper w35275 (MIT / Wharton), 'https://www.nber.org/system/files/working_papers/w35275/w35275.pdf' — for the identification strategy and construct definitions.
• incompleteThe app-store 'usage stayed flat' evidence is used to prove that AI-flooded software fails to find an audience, but the article does not isolate which released apps were AI-assisted or control for baseline market saturation, so the causal step from 'more releases + flat usage' to 'AI software attracts no audience' is asserted rather than demonstrated.
  AI isn't making developers more productive – it's making them busier — read next: The primary study's market-outcome methodology; app-market baseline data on release-to-usage ratios independent of AI adoption.
• incompleteThe 'north star of good' rests on a single, self-selected success case (Fin) whose own footnotes concede special conditions — a decade of in-house AI expertise and a near-death experience that 'cleared the deck' — and whose headline metric (merged PRs / R&D headcount) she admits is imperfect. How representative or reproducible the 3x result is for teams lacking those conditions is asserted ('well within reach for the rest of us') more than demonstrated.
  AI Enthusiasts Are in a Race Against Time, AI Skeptics Are in a Race Against Entropy — read next: Read controlled/longitudinal AI-productivity studies and the measuring-AI-impact literature on selection bias in success case studies and on outcome-anchored, oppositional metrics beyond PR counts.
• incompleteNo production evidence that the tool reduces MTTR or improves incident outcomes. The entire benefit case rests on toil removed (15-45 minutes of manual search, 'thousands of engineering hours lost per year') plus the assertion that responders can then 'skip straight to actual problem-solving' — but time-to-brief is not time-to-resolution, and nothing here is a controlled before/after or a field measurement. For a rigorous adopter this is the first missing thing: a build log's plausible mechanism, not demonstrated impact.
  Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident — read next: Incident-response outcome measurement: DORA's failed-deployment-recovery / MTTR data, controlled before-after studies of triage-automation on resolution time, and whether pre-assembled context actually shortens the diagnosis phase or just relocates the clock.
• incompleteThe four prescribed changes (AI Quality Gate role, defect-prevention-rate metric, written rationale, rotation) are asserted from a single team's experience. The piece reports they 'shifted how the team talked about the work' but offers no attrition, retention, or defect data demonstrating they reduce burnout or turnover -- the remedy is a plausible design, not a validated one.
  AI productivity is burning out your best engineers — read next: Controlled or longitudinal evidence on oversight-role rotation and validation-metric introduction versus engineer retention; before/after outcome measurement rather than testimonial.
• incompleteThe central remedy -- make the invisible visible via a 'defect prevention rate' -- presumes prevented incidents can be counted, but the piece gives no method for measuring an incident that did not happen or for attributing a catch to a specific engineer, leaving the metric without an operational definition and vulnerable to the same gaming and blindness it is meant to cure.
  AI productivity is burning out your best engineers — read next: Measurement of prevention and near-misses (safety science, SPACE framework); the 'busier not more productive' attenuation data on why activity proxies mislead.
• uninformedEvery quantitative claim in the piece traces back to Google's own blog post -- customer testimonials and vendor figures -- with no independent benchmark, replication, or third-party audit of any single number.
  Google's AlphaEvolve Reaches General Availability with Evolutionary Code Optimization as a Service — read next: An independent technical evaluation of AlphaEvolve GA performance, or a customer's own engineering blog post (e.g. from Klarna or JetBrains) corroborating the cited figures in more detail.
• incomplete'Throughput instead of estimation' is recommended as the system-observability metric, but the piece doesn't specify the level at which throughput should be measured (team? value stream? whole org?) or address the well-known failure modes of throughput-as-a-target shared by any WIP/flow metric — e.g. reclassifying, splitting, or gaming units of work to inflate apparent flow, the same gaming concern the piece itself raises about Tokenmaxxing.
  Reinventing team flow within AI uncertainty — read next: Look for flow-metrics literature (Kanban/DORA) on throughput-gaming countermeasures, and whether those apply unchanged in an AI-accelerated context.
• incompleteThe piece treats individual experimentation ('individual actions remain valid and don't require any upfront agreement') and team-owned experimentation (Popcorn Flow, assessed as a team) as compatible, coexisting levels, but never works out what happens when they conflict — e.g. an individual's unilateral habit change (dictation instead of typing prompts) versus a team's committed, board-tracked experiment that assumes a different interaction pattern.
  Reinventing team flow within AI uncertainty — read next: Look for material on reconciling individual experimentation with team-level process commitments more generally — where the boundary between 'my own workflow tweak' and 'something the team needs to agree on' actually sits.
• incompleteFowler asserts that unmeasurable functions like law, marketing, and education 'are managed' without explaining how — leaving the practical alternative to measurement (what replaces metrics in actual management decisions) completely unaddressed.
  CannotMeasureProductivity — read next: The DORA/SPACE/Core-4 tradition and this corpus's own 'metrics as feedback mechanism, not comparison tool' framing (extract-8-engineering-metrics-ai-broke) is arguably the field's 20-years-later answer: use flow/quality metrics as local feedback signals for a team rather than as cross-team/cross-individual productivity rankings — exactly the distinction Fowler gestures at but never develops.
• uninformedWriting in 2003, Fowler could not have anticipated generative AI's effect on code volume: AI-assisted development now produces measurably more code per unit of human effort (with reported ~20% larger AI-generated code than human-written code for equivalent tasks), which would further inflate LOC-based studies and make his critique more urgent, not less.
  CannotMeasureProductivity — read next: Directly extended by this corpus's AI-era engineering-metrics critiques (extract-8-engineering-metrics-ai-broke, extract-tbm-437-tokens-hours-points-proxies), which independently rediscover that easily-measured, easily-inflated proxies (LOC, tokens, story points, PR counts) systematically substitute for the still-unmeasured true output.
• incompleteThe headline 'AI's 2025 was cloud's 2010' analogy is asserted on Majors's own authority ('comfortable saying this') without engaging any disanalogies — cloud's 2010 mainstreaming was driven by quantifiable cost/reliability economics (published SLAs, TCO comparisons), whereas this piece's claim that AI coding reached a comparable threshold rests on anecdote and personal conviction, not the kind of adoption data Orosz himself has published elsewhere in this corpus (e.g., Cursor's two years of usage statistics).
  Stop being skeptical about AI for development with Charity Majors — read next: extract-pragmatic-engineer-cursor-ai-coding-stats (same author, quantitative data that could have grounded this analogy but isn't cited here).
• incompleteEverything is asserted from Anthropic-internal and community experience with no comparative or quantitative evidence: no A/B of CLAUDE.md-vs-none, no measured effect of subagents on context availability, no failure-rate or effect-size data for any recommended practice. It is impossible to tell which practices generalize versus which are idiosyncratic to Anthropic's workflows.
  Claude Code: Best practices for agentic coding — read next: Look for controlled Claude Code workflow comparisons and benchmark data (e.g. extract-cc-caveman-benchmark for token measurement); read against extract-charlie-hills-32-things-claude-code#g1, which flags the same evidence gap for a parallel tips list.
• incompleteThe article reports GitClear's headline percentages without the underlying methodology, so a reader cannot assess how each of the eight metrics is computed, whether they are de-biased, or how a commit is classified as 'AI-assisted' -- it is a short trade-press summary standing in for the primary study.
  Code maintainability plummets in the AI coding era — read next: Read GitClear's full 'AI Code Quality & Maintainability Gap' report and its metric definitions; compare against the earlier 2025 GitClear methodology it builds on.
• incompleteEvery headline figure is an era-level or temporal comparison (vs pre-AI, since 2023, relative to the base year) presented alongside AI's rise, not a controlled measurement of AI-authored versus human-authored code. The piece supplies a plausible mechanism but never isolates AI as cause versus confounds -- an influx of juniors, deadline pressure, growing codebase age, or shifting language mix could all move the same metrics.
  Code maintainability plummets in the AI coding era — read next: Look for studies that segment commits by authorship (AI vs human) and control for team seniority and codebase maturity before attributing the decline to AI.
• incompleteGitClear and GitKraken sell tooling into exactly this market, and no non-vendor telemetry is cited to triangulate the specific magnitudes. DORA is invoked for direction (instability rises with AI use) but does not corroborate GitClear's particular numbers, leaving the effect sizes reliant on a single interested source.
  Code maintainability plummets in the AI coding era — read next: Seek independent, non-vendor telemetry on AI code quality; see the corroborating-but-separate Faros AI dataset (22k developers) and DORA's own instability findings.
• incompleteThe human/AI division of labor is characterized qualitatively (AI does first-pass translation; humans supply schema, tests, and optimizations) but never quantified: no data on what fraction of methods the AI got right on the first pass versus needed manual fixes, how many iterations the loop took, or how the AI-driven migration compared in effort or time against a conventional hand-refactor.
  How Datadog Used Claude and Cursor for Test-Driven Production Migration — read next: Read the original Datadog engineering post (datadoghq.com/blog/engineering/ai-assisted-storage-migration) for any first-party figures the InfoQ summary dropped; then compare with extract-doordash-copilot-ios-test-migration#g1, which flags the same missing intervention-fraction data for an AI-assisted test migration.
• incompleteThe paper explicitly cannot distinguish whether flat marketplace usage reflects lower-quality marginal apps or a consumer-side discovery bottleneck, but it doesn't engage with a third possibility its own data seems to hint at: that lowering the fixed cost of producing a 'minimally publishable' app may simply attract projects (personal tools, internal experiments, portfolio pieces) that were never intended to find a broad audience, and that counting these as market 'supply' in the first place may be the wrong frame.
  Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools — read next: Look for research that segments new app releases by evident intent (commercial vs. personal/experimental) rather than treating all new listings as attempts to reach a broad audience.
• uninformedThe paper explicitly excludes enterprise and internal-only software — likely where most professional, team-based software engineering work actually happens — from its analysis. Its production-hierarchy model and 0.25 elasticity estimate, derived from public-repository and consumer-marketplace data, may not transfer to internal systems with different review norms, continuous-deployment cadences without a discrete 'release' artifact, and different definitions of 'shipped.'
  Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools — read next: Look for internal, enterprise-scale telemetry studies (e.g. DORA's survey-based team archetypes) to see whether the same attenuation/elasticity pattern holds inside organizations rather than on public GitHub and consumer app stores.
• incompleteThe empirical base is thin for a strong, society-wide generational claim: one 319-worker Microsoft/CMU study (a correlation between AI trust and critical thinking, read as causal) plus the author's own course anecdote.
  Det skal være vanskelig — read next: Longitudinal studies of learning outcomes under AI-assisted vs. traditional instruction; the Dreyfus skill-acquisition model; replication of the Microsoft/CMU finding and its direction of causation.
• incompleteThe article measures speed to a PR milestone but never addresses whether AI-accelerated onboarding produces developers who understand systems as deeply as those who onboarded more slowly. Faster PRs could mean shallower understanding, higher downstream rework, or increased dependence on AI as a crutch rather than developing genuine codebase knowledge. The article acknowledges this gap but treats it as a future research item rather than a fundamental limitation of the current finding.
  Developer ramp-up time continues to accelerate with AI — read next: Look for research on knowledge retention and long-term developer effectiveness when onboarding is AI-assisted vs. traditional. The Dreyfus skill acquisition model and research on desirable difficulties in learning suggest that compressing the 'figuring things out' phase may have hidden costs.
• incompleteThe cross-sectional comparison from Q4 2025 (49 days with AI vs. 91 days without) suffers from the exact selection bias DX/Abi Noda has warned about in other publications: engineers who adopt AI daily may be systematically different from those who do not. The Q1 2026 data shifts to a longitudinal comparison (same cohort, AI users, tracked over time) which is methodologically better, but the headline '50% reduction' still rests on the weaker cross-sectional design.
  Developer ramp-up time continues to accelerate with AI — read next: DX's own meta-longitudinal study (referenced in the companion podcast extraction); academic literature on selection effects in technology adoption studies.
• incompleteNo discussion of what 'daily AI use' means operationally. Is it measured via tool telemetry (Copilot usage logs), self-report, or organizational policy? The definition matters enormously for interpreting the results -- if it is policy-based ('we tell everyone to use AI'), it conflates organizational onboarding quality with AI-specific effects.
  Developer ramp-up time continues to accelerate with AI — read next: DX methodology documentation; compare with how other studies define and measure AI adoption intensity.
• uninformedThe article does not consider that the PR bar itself may be shifting. If organizations are breaking work into smaller PRs (a common practice enabled by AI-assisted coding), then 10th PR arrives sooner not because the developer is ramping up faster but because the unit of work has shrunk. Without controlling for PR size or complexity, the metric is ambiguous.
  Developer ramp-up time continues to accelerate with AI — read next: Research on PR granularity trends in AI-assisted development; DX's 'true throughput' metric which weights by lines of code.
• incompleteThe article asserts a mechanism (senior developers 'see the bigger picture' and 'prompt more effectively') for why AI amplifies seniority, but the underlying GitClear study as described only measures output metrics (Diff Delta, commits, PRs) longitudinally — it does not report any direct measurement of prompting behavior, code review quality, or 'big picture' understanding that would actually test the proposed mechanism rather than merely restate the correlation in explanatory language.
  AI doesn't create great developers, it amplifies them — read next: Research that directly measures prompting strategy or decision quality (rather than output volume) across experience levels would be needed to confirm the proposed mechanism rather than just the correlation.
• incompleteThe piece uses causal language ('drive better outcomes') for the capability-outcome relationship but never states DORA's research design in enough detail (survey design, longitudinal vs. cross-sectional, controls) to know whether that relationship is measured causally or is a correlational finding from self-report survey data — a distinction this reader's stated preference for 'benchmark data over vendor claims' would specifically flag.
  AI-Assisted Software Development: Team Profiles and Capabilities for Putting Research into Action — read next: DORA's published methodology section, or the AI Capabilities Model report (December, referenced in the article) for the underlying statistical treatment.
• incompleteThe article surfaces the counterintuitive quality-up/instability-up finding and Harvey's own hypothesis for it, but never reports whether DORA tested that hypothesis against the data (e.g. correlating instability with specific missing capabilities) — it's presented as plausible interpretation, not a verified mechanism.
  AI-Assisted Software Development: Team Profiles and Capabilities for Putting Research into Action — read next: DORA report sections specifically on drivers of delivery instability, or a follow-up interview probing this finding.
• empiricalThe 10-15% throughput gain figure is stated without methodology detail. How were confounders controlled? What is the variance across organizations? The meta-longitudinal study is referenced as forthcoming but not yet available.
  Measuring AI impact, assessing readiness, and new data trends — read next: Look for DX's published meta-longitudinal study and methodology; compare with academic studies on AI-assisted development.
• conceptualThe amplification/augmentation framework measures output volume (throughput, human-equivalent hours) but does not address output quality, defect rates, or long-term maintainability of AI-generated code.
  Measuring AI impact, assessing readiness, and new data trends — read next: Investigate literature on technical debt and defect rates in AI-assisted codebases; check whether DX's framework has quality dimensions not discussed here.
• structuralThe discussion treats AI readiness as a developer-experience problem but does not address organizational incentive structures -- e.g., why documentation and CI remain chronically underinvested despite being acknowledged as important for decades.
  Measuring AI impact, assessing readiness, and new data trends — read next: Organizational economics of developer tooling investment; why 'table stakes' remain unmet.
• perspectivalThe entire conversation is from the perspective of engineering leaders at large, well-resourced companies (Twilio, Booking.com, Block). The experience of small teams, non-tech-native companies, or individual contributors is absent.
  Measuring AI impact, assessing readiness, and new data trends — read next: Seek practitioner reports from smaller organizations and IC perspectives on AI adoption.
• incompleteThe article juxtaposes studies with materially different methodologies, populations, task types, and time windows (2023 vendor-run benchmark on a toy HTTP-server task; 2024 enterprise field study; 2025 METR RCT on real OSS repos; 2025 Faros telemetry) as though they measure the same underlying quantity ('AI coding productivity'), without discussing whether the specific percentages are actually comparable across such different conditions.
  The productivity paradox of AI coding assistants — read next: Any methodological meta-analysis of AI-coding-productivity studies, if one exists in the corpus or elsewhere, would test whether this juxtaposition actually holds up.
• incompleteThe post claims release frequency is 'the only number that matters' but doesn't argue why — and Lean's own toolkit would suggest cycle time, quality, and customer outcomes belong alongside release frequency. The reduction of 'real value' to a single metric is rhetorically clean but theoretically thin.
  Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard — read next: DORA's four-key-metrics literature; Reinertsen on cost of delay; Steve Yegge / Gene Kim on quality-vs-speed under AI; Beck's recent newsletters on what to measure when AI changes throughput.
• incompleteTheodo's consultancy context is a selection bias engine that Bernhard doesn't address. Consultants typically work on greenfield engagements or on well-bounded modernisation projects where business needs are pre-scoped by a client brief. The very bottlenecks Bernhard names (unclear business needs, organisational dependencies) are precisely the bottlenecks consultancy contracts pre-filter for. Whether the 3x result transfers to in-house teams operating in the muck is the load-bearing open question.
  Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard — read next: Compare to Anthropic enterprise PDF findings on in-house deployments; Cutler on internal-team organisational drag; case studies of in-house legacy modernisation at scale (e.g. NAV, Entur reports already in corpus).
• incompleteLegacy modernisation is presented as 'a' bottleneck-aligned use case but Bernhard doesn't enumerate what makes it bottleneck-aligned and what other use cases would qualify. A more complete argument would name the properties (translation-heavy, human-comprehension-bound, well-specified target, measurable acceptance criteria) that make AI bottleneck-effective.
  Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard — read next: Mary Poppendieck on Lean software value-stream mapping; Theory-of-Constraints applied software literature; specific case studies of AI-driven greenfield vs migration projects.
• incompleteThe post takes the MIT/UPenn paper as authoritative but doesn't disclose the dataset, sample, or timeframe. Async agents in 2024-25 (the likely study window) are not the same product category as 2026 long-running harnesses — the 1.3x number may understate what later-generation tooling can do at the release-frequency end, OR may be optimistic if the study captured the early enthusiasm phase.
  Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard — read next: The Demirer/Musolff/Yang paper itself; recent METR / RCT-style AI productivity studies; Pocock workshop on what changes with harness-level tooling.
• incompleteThe 'we generate better learnings' claim is asserted without describing Theodo's actual learning mechanisms — internal write-ups, retros, replication across client engagements, etc. For an audience interested in adopting the approach, the learning infrastructure is the load-bearing part.
  Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard — read next: Theodo's engineering blog (theodo.com / padok.fr); Mike Rother's Toyota Kata on learning routines; A3 thinking literature.
• uninformedThe post predates / doesn't engage with the harness-engineering thread that this corpus has been tracking (Anthropic harness paper, Pocock workshop, Cloudflare Mythos). Harness-level tooling specifically aims at the integration between code generation and release — exactly the gap Bernhard names. Whether harness engineering is a third path (between 'roll out Copilot' and 'consultancy-led bottleneck targeting') is an obvious adjacency.
  Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard — read next: Pocock workshop extraction; Anthropic harness design paper; Cloudflare Mythos extraction; the Anthropic enterprise PDF extraction.
• incompleteThe report repeatedly argues a causal story (AI adoption drives quality decline) from correlational, cross-company telemetry (Spearman's rho at the team-quarter level) -- the same methodological gap this corpus's extraction of the DORA AI Capabilities Model report (extract-dora-ai-team-profiles-and-capabilities#g2) flags as unaddressed there. Confounds such as simultaneous process changes, hiring waves, or deadline pressure occurring in the same high-adoption quarters are not ruled out.
  The Acceleration Whiplash — read next: A controlled or quasi-experimental design (staggered rollout comparison, matched teams) isolating AI adoption from co-occurring organizational changes.
• incompleteConvergence is asserted, not tested. The five studies use different methods and measure different constructs (PR throughput, release counts, DevEx dimensions, a tooling wishlist, a debt taxonomy), yet their 'convergence' is asserted by the curator, and the disclosure that three of five come from close colleagues is noted but never reconciled with the strength placed on their agreement.
  Five studies that are changing how I think about AI in software engineering — read next: Read against the same construct-agreement worry raised elsewhere in the corpus: extract-agentic-code-review#g1 questions whether 'independent datasets that agree' truly measure one underlying thing when the metrics differ. Apply that skepticism to whether these five studies measure one phenomenon.
• incompleteNo independent evidence that released capability translates into changed real-world work. Every 'how you work changes' claim rests on benchmark numbers, vendor demos, and enthusiastic X reactions — no controlled task studies, no failure-rate or override data on real workflows.
  How the 4 New AI Models Change How You Work — read next: Read against work that separates real AI productivity effects from adoption theater — longitudinal, oppositional, outcome-anchored measurement — and against the hype critique.
• incompleteThe article treats AI as equivalent to a junior developer, but AI has fundamentally different failure modes — it can be confidently wrong in ways juniors rarely are, and it lacks the ability to say 'I don't understand the requirement.' The junior-developer analogy may lead to under-investment in verification.
  Patterns for Reducing Friction in AI-Assisted Development — read next: Research on AI confabulation/hallucination in code generation; studies comparing AI and human error patterns in software development
• incompleteThe framework assumes the developer can articulate their conventions and constraints explicitly enough for AI to follow. This is Polanyi's paradox — we know more than we can tell. The Sensible Defaults pattern may hit a ceiling where the most important quality signals are precisely the ones hardest to codify.
  Patterns for Reducing Friction in AI-Assisted Development — read next: Polanyi's 'The Tacit Dimension'; research on knowledge management and the limits of codifying expert judgment
• incompleteNo discussion of how these patterns interact with rapidly improving AI capabilities. If AI models become better at inferring project conventions from code context, some of these patterns (especially Knowledge Priming) may become unnecessary. The framework lacks a theory of its own obsolescence.
  Patterns for Reducing Friction in AI-Assisted Development — read next: Papers on in-context learning improvements in LLMs; research on retrieval-augmented code generation
• incompleteThe article focuses entirely on individual developer-AI interaction. There is no treatment of how AI collaboration patterns interact with team dynamics — e.g., whether standardized prompts reduce healthy variation in approach, or whether shared priming docs create new coordination costs.
  Patterns for Reducing Friction in AI-Assisted Development — read next: Research on coordination costs in software teams; Brooks's 'The Mythical Man-Month' on communication overhead
• uninformedNo engagement with the existing literature on human-AI teaming from other domains (aviation, medicine) where collaboration frameworks have been developed and empirically tested for decades.
  Patterns for Reducing Friction in AI-Assisted Development — read next: Research on crew resource management (CRM) in aviation; human-AI teaming frameworks from DARPA and military contexts
• incompleteNo evaluation of whether these UI additions actually improve developer outcomes. The flow/reduced-context-switching benefit is asserted with no usage, adoption, or before/after productivity data.
  GitHub Copilot CLI Gets Tabs and No-Config-File Tool Setup in Redesigned Terminal UI — read next: Compare with extract-copilot-cli-general-availability#g2 (the same missing-benchmark gap for the product's GA) and measuring-ai-impact-style outcome-anchored evaluation; a feature-add announcement is not evidence of a productivity effect.
• incompleteNo error rate, override rate, or analyst-correction rate is reported for the 'self-service' answers that increased from 53-90% depending on request type — a rising self-service percentage says nothing about whether those answers were correct, unless paired with a quality metric.
  Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents — read next: Look for a companion piece or the original Grab blog post's methodology section on how self-service answer quality is audited or sampled; compare against the HBR piece's argument that self-service analytics increases (not decreases) the need for metric governance.
• incompleteThe only evidence offered that telemetry-grounded verification actually improves outcomes is a single vendor-sourced internal anecdote (the Tempo squad) and a hypothetical worked example (the payment provider). No data compares defect rates, review time, or actual reviewer engagement between dashboard-linked PRs and plain diff-plus-description PRs. Given that the article's own diagnosis is that reviewers already rubber-stamp ('LGTM syndrome'), it is not shown that a dashboard link gets scrutinized any more carefully than a diff did — the same failure mode could simply migrate one layer up.
  Grafana's gcx and MCP Server Reach GA for Telemetry-Driven Agent Development — read next: Independent, non-vendor telemetry on whether richer PR context (dashboards, traces) measurably changes reviewer behavior or defect escape rates; the corpus's existing LGTM-syndrome and PR-review material.
• incompleteNo methodology is given for isolating AI's causal, incremental contribution to the reported outcomes (Aramco's $2.6B, Netflix's ~$1B, Merck's >50% reduction, Nike's 20%/15%/10% figures) from other concurrent initiatives, macro conditions, or reporting incentives. All figures appear self-reported by the companies, their vendors, or press coverage, with no control-group or counterfactual comparison — the same 'vendor self-reported metrics' pattern flagged elsewhere in the corpus.
  Closing the Gap Between AI Investment and Financial Return — read next: Read James Shore's AI-ROI series (extract-shore-ai-money-worth-part1) for a methods-focused critique of anecdotal and self-reported AI impact claims, and how to demand rigor before trusting a number.
• incompleteThe article never explains how 'adoption' and 'strategic alignment' interact: if a technically excellent, strategically well-aligned AI solution fails to be adopted, is that an independent adoption failure, or does poor strategic targeting itself suppress adoption because employees can tell the problem doesn't matter? The equation treats the three factors as independent multiplicands, but the 'one minute saved' example suggests strategic relevance may itself drive whether people bother to adopt a tool.
  Closing the Gap Between AI Investment and Financial Return — read next: Read James Shore's AI-ROI series on the change-management cost of adoption (extract-shore-ai-money-worth-part1) to test whether adoption and strategic alignment are genuinely separable factors or entangled.
• incompleteThe 'moat' claim does not address that the article itself publishes the general four-step method to a mass HBR audience, nor does it engage with why competitors following the same publicly available method wouldn't erode the described advantage. It also never distinguishes alignment-as-analysis (replicable by any competent competitor reading this article) from alignment-as-proprietary-data (a genuine, durable moat).
  Closing the Gap Between AI Investment and Financial Return — read next: Read strategy literature on sustainable competitive advantage / the resource-based view to test whether 'knowing your own economics' is actually rare or inimitable once the method for finding it is public.
• incompleteThe piece never addresses that its authors are Perplexity staff studying engagement with, and the economic value of, Perplexity's own commercial assistant and agent products — a direct conflict of interest disclosed only as a parenthetical ('where the author team works') and never discussed as a limitation, even though the itemized limits paragraph addresses several other caveats in detail.
  Research: How AI Agents Broaden the Scope of Knowledge Work — read next: An independent, non-vendor field study of assistant-vs-agent usage — academic or from a researcher with no commercial stake in either product — to see whether the scope-expansion effect replicates outside a single company's own telemetry.
• illogicalThe 'second, LLM-based estimate' is framed as independent triangulation ('we didn't rely on a single method') that corroborates the primary human-equivalent-cost estimate, but both estimates were produced by the same research team using models and assumptions of their own choosing; agreement between two internally-generated estimates is much weaker evidence of accuracy than the framing implies, since it cannot rule out a bias shared by both methods.
  Research: How AI Agents Broaden the Scope of Knowledge Work — read next: A genuinely external replication — a third party re-deriving human-equivalent time/cost from comparable session logs using an independently chosen costing methodology.
• incompleteThe authors recommend routing agent work by 'complexity and verifiability' and building a human commit boundary for consequential actions, but their own scope data shows agent use crossing further outside users' primary occupational expertise over time — the exact condition that makes verification hardest — and the piece never measures whether cross-occupational agent output is actually being verified, or is simply being accepted on trust.
  Research: How AI Agents Broaden the Scope of Knowledge Work — read next: Research on whether and how non-expert users actually catch errors in AI output outside their own domain (calibration and overconfidence in cross-domain review), which the piece assumes a verification capacity for but never tests.
• uninformedThe definitions behind 'higher-order cognition,' 'create-level work,' 'abstract vs. routine,' and 'more complex' unlocked tasks are never operationalized — no coding scheme, no example task placed in each bucket, no inter-rater reliability or human-audit check — so a reader cannot judge how much of the reported scope-expansion is real behavioral shift versus an artifact of how the classifying LLM was prompted.
  Research: How AI Agents Broaden the Scope of Knowledge Work — read next: The underlying research methodology or appendix (if published separately from this HBR summary) for the actual classification scheme and any reliability checks performed.
• incompleteNo outcome-level metrics are ever supplied for Wayfair — the only figures given (2.5 million tags corrected, 41,000 supplier tickets automated per month) are explicitly volume/activity metrics, which is precisely the kind of metric the article's own "optimize the wrong metric" failure mode warns against trusting. The authors state that "those figures demonstrate the scale of the redesigned workflow, not by themselves its business value," but never return with the actual conversion, return-rate, or supplier-resolution-quality numbers that would establish Wayfair as a genuine success rather than merely a large-scale one.
  Stop Automating Old Processes. Design New Ones Instead. — read next: Primary Wayfair/OpenAI case-study material with before/after conversion, return-rate, and supplier-resolution outcome data, to check whether the redesign actually cleared the article's own outcome test.
• illogicalThe piece argues both that AI's pace of change is unlike anything the author has seen in 20+ years, and that established change-management patterns from prior technology waves (DevOps, cloud, CD) will straightforwardly transfer — without reconciling why a genuinely unprecedented shift would be well-served by a non-unprecedented adoption playbook. The claim that 'proven methodologies' suffice is asserted, not argued against the claim of novelty made two paragraphs earlier.
  Leading your engineers towards an AI-assisted future — read next: Look for arguments specifically about what, if anything, is structurally different about AI adoption that would break a Mission Command-style rollout — e.g. the rate of tool obsolescence outpacing a Community of Practice's ability to stay current.
• incompleteThe paper cannot separate token-volume growth driven by more/deeper human-initiated usage from token-volume growth driven by model-generation changes — e.g., longer reasoning traces or more verbose outputs per query as GPT models shifted toward chain-of-thought reasoning across the 2025-2026 sample window. Output tokens per message could rise mechanically even with roughly flat human engagement, inflating the headline 'sevenfold' and 'fourfold within-cohort' growth figures independent of any real deepening of organizational deployment.
  How Organizations Use AI: Evidence From ChatGPT — read next: Documentation or research on how output-tokens-per-query changed across OpenAI model generations over this period, or a robustness check using message counts (rather than tokens) as the primary growth metric — the paper does report messages elsewhere but leads with tokens for its headline growth claim.
• illogicalThe financial usage-intensity analysis (Figure 4, Table 2) is careful in prose to disclaim causality, but its visual and rhetorical presentation ('high-intensity adopters are shifted to the right') still invites readers to treat usage intensity as doing genuine explanatory work for firm value, without seriously entertaining the confound that firms with cash-rich, technically sophisticated, high-margin workforces would plausibly show both higher revenue/market-value per employee AND higher AI usage intensity for reasons unrelated to any causal effect of AI itself (e.g., general workforce quality or industry structure).
  How Organizations Use AI: Evidence From ChatGPT — read next: A paper or extension using instrumental variables, a natural experiment (e.g., staggered enterprise rollout timing within a firm), or panel fixed-effects design to isolate whether usage intensity has any causal financial effect, rather than firm-level cross-sectional correlation.
• incompleteMethodological (the study's own thinnest point): 'social listening' on public posts is a convenience sample of vocal, self-selecting users. There is no denominator, no correction for selection bias, and no way to distinguish frequency-of-posting from frequency-of-use — so a ranking of what people TALK ABOUT online is presented as a ranking of what people DO, and emotionally charged uses (therapy, grief, romance) that generate more posting are structurally over-weighted. The order-of-magnitude larger dataset improves precision but not representativeness.
  How People Are Really Using AI in 2026 — read next: Read the author's own 'Making sense of research on how people use AI' (HBR, Nov 2025) alongside survey- or telemetry-based usage studies (e.g., first-party Anthropic/OpenAI usage reports) to triangulate the social-listening picture against a representative sample and quantify the selection bias.
• incompleteConsumer/individual usage is used to reason about enterprise value: the dataset is 'mostly individuals... at home and at work,' yet the business transformation verdict is drawn from it. Individual bottom-up use and organizational value are different units and are conflated, which both flatters adoption breadth and muddies the value verdict.
  How People Are Really Using AI in 2026 — read next: Read extract-who-cares-about-consumer-ai (the ~100x per-user token-economics split between consumer and enterprise) and enterprise-specific ROI studies to keep consumer-scale adoption separate from organization-level value realization.
• uninformedThe thinkslop harm is asserted at population scale from vivid single-user anecdotes. The study measures the prevalence of use cases where thinking COULD be outsourced (a quarter of top uses) but not the cognitive damage actually incurred, and — crucially — it presents the sharpening 'sparring-partner' use from the same data without ever quantifying the crutch-vs-foil ratio, so the headline concern rests on an unmeasured mix.
  How People Are Really Using AI in 2026 — read next: Read the cited 'cognitive debt' study (arXiv 2506.08872, 'Your Brain on ChatGPT') and extract-cognitive-surrender (offloading vs. surrender, posture-dependence, the 73% wrong-answer acceptance finding) for the mechanism and the controlled evidence the HBR piece substitutes anecdote for.
• incompleteNo success metrics, SLAs, or independent measurement framework are defined for the program — no stated target for vulnerabilities remediated, mean-time-to-fix, patch-acceptance rate upstream, or third-party audit — leaving 'help enterprises secure open source software' effectively unfalsifiable.
  IBM and Red Hat Commit $5 Billion to Redefine the Future of Open Source in the AI Era — read next: Compare against how the corpus's Project Glasswing extraction (Cloudflare/Mythos) is criticized for the same missing-baseline problem at a technical level; look for whether Lightwell publishes any comparable data over time.
• incompleteThe analogy between Bainbridge's control-room operators and software engineers is taken as given but never defended. Bainbridge studied operators of physical, bounded, regulated, slow-dynamics systems (nuclear plants, chemical processes, aviation cockpits) where (a) the human's role is fundamentally MONITORING-AND-OVERRIDE, (b) the system's correct behavior is specified ex ante, (c) regulators audit the human-machine boundary, and (d) the operator is licensed and trained on simulators. Software engineers are CONSTRUCTORS — they design the system itself, not just operate it; the system's correct behavior is partially undefined; no licensing or regulator polices the boundary; and the 'incident' role is only one slice of the work. Reed's mapping works best for the incident-response slice (which is what his job is) and is weakest for routine development. He never disambiguates these. The argument that 'AI ironies for engineers = Bainbridge ironies for operators' may overgeneralize from the incident-response edge case to the whole of engineering practice.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Read the original Bainbridge 1983 paper directly. Then Hollnagel & Woods on resilience engineering, and David Mindell's 'Our Robots, Ourselves' (history of human-machine collaboration). Pair with Naur's 'Programming as Theory Building' for the contrasting view that software engineering is theory construction, not operation.
• incompleteReed's prescription assumes organizations will reinvest AI-driven efficiency gains in human thoroughness, but the same talk reports that the current pattern is the opposite: 'AI's main impact on my life is more incidents because we're generating more code.' The systems-thinking question of how to design feedback loops that prevent capture of efficiency-as-throughput is left untouched. Without that loop, the 'AI does efficiency, humans do more thoroughness' division is a moral exhortation, not a mechanism.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Donella Meadows on leverage points (especially feedback loops, rules of the system, and goals of the system). John Cutler on institutional overload and on how organizations metabolize efficiency gains. Possibly Christensen on capabilities-as-process — i.e., the org's existing process consumes new capability.
• incompleteThe empirical evidence is thin and uneven. The nurses study (Woods et al., July 2025) is presented as decisive but is a single study, in a different domain (clinical prediction), with a likely small N; the developer study (METR-style) is cited from memory. Reed does not engage the contrary evidence (e.g., positive AI productivity findings from DORA, GitHub Copilot studies) or distinguish task types where AI does and does not pay off. The asymmetric-risk frame is compelling and probably right but is underdetermined by the cited data.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: DORA 2024/2025 AI findings, the Princeton/MIT studies on developer AI productivity, METR's actual published methodology, Woods's published paper (rather than the slide). Also Cal Newport / Bainbridge-skeptics on whether the operator analogy fits knowledge work.
• incompleteThe animacy paradox is introduced as a phenomenological observation but is not connected to design recommendations. If interpretation flips with situational stakes, what design pattern reliably keeps the deterministic-machine interpretation alive in high-tempo contexts? Reed gestures at attribution and explainability but doesn't develop this.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Lucy Suchman 'Plans and Situated Actions'; Dave Woods's papers on 'envisioned worlds' and on coordination breakdowns; possibly the air-traffic-control HCI literature on 'mode confusion'.
• incompleteReed touches the systems-thinking question (juniors who never built the mental model, attrition of senior engineers carrying tacit knowledge) but doesn't push it to its leverage-point: the org's HIRING and TRAINING flows may be where the irony actually lives, not in the engineer's individual relationship to AI. The dynamic he describes is a stock-and-flow problem (the stock of tacit knowledge is depleting faster than it is being replenished) and warrants stock-and-flow analysis.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Meadows again on stocks-and-flows; the National Research Council's literature on knowledge attrition in nuclear and aerospace; and possibly the apprenticeship literature (Lave and Wenger, situated learning).
• illogicalReed conflates 'AI is non-deterministic' with 'AI lacks interpredictability.' These are related but distinct. Humans are non-deterministic and have high interpredictability with their teammates. The reason AI lacks interpredictability is not non-determinism per se but lack of a SHARED MODEL of context, goals, and norms. This matters because it suggests the fix is not 'make AI more deterministic' but 'invest in shared context and norm-conformance' — a different research and product direction.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Herbert Clark on common ground in conversation; the multi-agent systems literature on shared mental models; recent work on AI alignment through context-engineering.
• uninformedNearly every external statistic cited (GitHub Copilot's $600M revenue and 15M users, Gartner's 2028 market forecast, AWS and Microsoft named as workshop partners, Anthropic CEO's 90%-AI-generated-code prediction) comes from a vendor or analyst with a direct commercial stake in agentic AI adoption, and the memo does not discuss this conflict of interest or seek independent corroboration.
  Agentic AI in the Software Development Lifecycle (SDLC) — read next: extract-ai-daily-brief-chatgpt-work-agent#p9 (vendor-reported productivity claims treated as adequate evidence) names the same general pattern this memo instantiates.
• illogicalThe headline claim that daily AI tool use correlates with the highest productivity and satisfaction is presented without addressing an obvious alternative explanation: developers who are already productive and satisfied may simply be more likely to adopt and stick with AI tools (reverse causation), or seniority/role/tech-stack may confound both variables. The paper's own Limitations section discusses external, construct, and internal validity but never raises this causal-direction threat specifically for the AI-usage finding.
  Time Warp: The Gap Between Developers' Ideal vs Actual Workweeks in an AI-Driven Era — read next: A longitudinal or quasi-experimental design (e.g., staggered AI-tool rollout) that tracks the same developers' satisfaction/productivity before and after adoption would distinguish cause from selection.
• incompleteThe quantitative analysis never splits results by role or seniority (e.g., team lead vs. individual contributor), even though the Discussion explicitly flags that team leads' workloads differ structurally from ICs' and that automating a satisfaction-depressing task for one role group might not generalize to another.
  Time Warp: The Gap Between Developers' Ideal vs Actual Workweeks in an AI-Driven Era — read next: A follow-up stratified by role/level would test whether the 'automate PR review, monitoring, communication, mentoring' recommendation holds for team leads as well as ICs, or backfires by removing work that leads specifically value.
• incompleteThe paper assumes that automating a task developers currently want automated (documentation, environment setup) will reduce the time they spend on that category, but corpus evidence elsewhere (e.g., AI shifting developer effort from code-writing to code-reviewing) suggests automation often relocates effort to verification/correction rather than eliminating it. The paper never considers that documentation or test automation could produce the same verification burden it hopes to avoid.
  Time Warp: The Gap Between Developers' Ideal vs Actual Workweeks in an AI-Driven Era — read next: Pair with field studies of AI documentation/test-generation tools in production to see whether time-on-category actually drops or merely changes character.
• incompleteThe central 'why lawyers' evidence — that lawyers are the deepest, most-returning users among knowledge workers — comes solely from Anthropic's own product lead (Pike), relayed second-hand in a newsletter that closes with a paid-workshop upsell. No independent usage data or cross-profession comparison is offered, so the load-bearing claim is an unverified vendor assertion.
  Lawyers are quietly becoming builders — read next: Independent, third-party AI-usage studies across professions; legal-AI adoption surveys not published by a model vendor; measured outcomes rather than depth-of-use anecdotes.
• incompleteThe presentation treats 'leadership practices' as the causal variable explaining per-company variance, but does not control for confounders like codebase quality, technical debt, team size, or domain complexity. Companies that already have good DevEx practices may simply be better at everything, including AI adoption.
  Presentation: Leadership in AI-Assisted Engineering — read next: Research on organizational readiness factors for technology adoption; look at DX's own AI Impact Report for more controlled analysis
• incompleteThe argument against AI replacing engineers relies heavily on current SWE-bench scores and a claim about algorithms not changing since the 1950s. This understates the pace of agentic and scaffolding innovations (MCP, RAG, multi-agent systems) that improve effective capability without changing underlying model architecture.
  Presentation: Leadership in AI-Assisted Engineering — read next: Anthropic, OpenAI, and DeepMind research on agent scaffolding; François Chollet's critique of benchmark-as-capability proxies
• incompleteDX is both the research organization and the vendor selling the measurement platform. The talk acknowledges DX's role but does not address how this commercial interest might shape which metrics are recommended or how 'success' is defined.
  Presentation: Leadership in AI-Assisted Engineering — read next: Independent developer productivity research (e.g., DORA team at Google, academic DevEx studies not commercially affiliated)
• uninformedThe claim that 'underlying algorithms haven't changed since the 1950s' is misleading. While perceptrons date to the 1950s, transformers (2017), RLHF, constitutional AI, chain-of-thought, and scaling laws represent genuine algorithmic innovations. The framing conflates 'neural networks exist since the 1950s' with 'nothing has changed.'
  Presentation: Leadership in AI-Assisted Engineering — read next: Vaswani et al. 'Attention Is All You Need' (2017); Kaplan et al. scaling laws papers; survey of post-transformer architectural innovations
• incompleteThe central internal tension: Elena diagnoses the theater as structurally incentive-driven (attention economy, VC→exec→employee cascade, career self-protection) yet prescribes a voluntary remedy — be honest, hold creators accountable, don't amplify untested claims. She appeals to individual virtue against an incentive gradient she herself says rewards hype, without any mechanism (verification infrastructure, benchmarks, changed rewards) to make honesty the dominant strategy. This is a collective-action problem treated as a matter of conscience.
  Please stop the AI Confidence Theater — read next: Costly-signaling and mechanism-design (Spence's job-market signaling), tragedy-of-the-commons remedies (Ostrom on governing shared resources — here, shared trust), and John Cutler on designing organizational conditions/incentives rather than exhorting behavior; connects to the persona's systems-thinking lens and 'organizational adoption of ideas' hidden layer.
• incompleteThe 'next thousand prompts' / living-systems point — that real value is in monitoring, evaluation, iteration, and tuning — is asserted but not operationalized. The essay offers no framework for how to actually measure honest AI value (evals, benchmarks, drift detection), which is precisely the verification infrastructure its argument implies is needed.
  Please stop the AI Confidence Theater — read next: LLM evaluation practice, DORA's AI findings on measuring engineering impact, and disciplined benchmark design — directly relevant to the persona's rigorous-adopter interest in what AI tools actually do to productivity and quality.
• incompleteThe hiring prescription ('work trials and case studies become absolutely necessary') ignores that work trials are costly, hard to scale, and can themselves be AI-assisted or gamed — the same forgeability problem may simply move one level up. The claim that costly demonstration restores signal is under-examined.
  Please stop the AI Confidence Theater — read next: Signal-design literature on hiring and evaluation under cheap-talk conditions; how costly signals degrade when the cost itself can be outsourced to AI.
• incompleteThe advice to tailor DevX measurement to whatever leadership already talks about (market share, margin, velocity) treats leadership's framing purely as a communication target, without addressing the risk that the framing itself could be distorted by AI hype or short-term pressure -- in which case tailoring measurement to match it risks reinforcing the distortion rather than surfacing the truth.
  How to measure AI developer productivity in 2025 | Nicole Forsgren — read next: extract-faros-acceleration-whiplash's distinction between self-reported survey data and telemetry-revealed costs is a useful corrective lens here -- it shows concretely how leadership-facing metrics (e.g. perceived productivity) can diverge from what instrumentation reveals.
• uninformedNone of the productivity, quality, or velocity claims (PMs/designers 'getting farther' before engineering unlocks things, 'much higher velocity, much higher quality' data analysis) are quantified — no benchmark data, before/after metrics, or even qualitative case counts are offered, despite this being exactly the kind of claim the reader's persona treats skeptically absent data.
  Netflix CPTO on AI and the future of product and tech roles | Elizabeth Stone — read next: DORA's 2025 AI Capabilities Model (extract with id dora-ai-capabilities-model territory) or any org's published before/after metrics on AI-assisted product development would let these claims be checked against measured outcomes rather than anecdote.
• incompleteMarcus equates 'pilot study RoI is poor' with 'AI investment lacks RoI' without distinguishing infrastructure plays (compute, model training as platform) from end-user productivity studies. Hyperscaler capex logic is not pilot logic.
  Could generative AI turn out to be the tech industry's Vietnam? And could public backlash lead AI to a better path? — read next: Erik Brynjolfsson's productivity-paradox work; recent Goldman/MIT/McKinsey AI ROI studies in tension with each other
• unsupported_claimThe title claims 'the model doesn't matter anymore' but the email body never argues or even mentions this claim. The provocative thesis is pure clickbait with no supporting content.
  The model doesn't matter anymore – and other AI skills worth your Thursday
• missing_evidenceThe email provides no evidence that the speakers or sessions will deliver on the 'signal from real practitioners' promise. Speaker credentials are mentioned only for one session (Paige Bailey from Google DeepMind).
  The model doesn't matter anymore – and other AI skills worth your Thursday
• incompleteNearly every quantitative claim (2x code quality, 47% more productivity from AI review, 70% of PRs high-severity, ~80% no human review, 88% context-attribution) comes from vendor-run or vendor-adjacent surveys — Qodo's own plus Sonar and Faros — sampled from 'at least half of our [1,000+] clients', i.e. self-selected AI-tool users, n≈617, with no independent baseline or controlled non-AI comparison. The figures are correlational and commercially interested, not causal measurement.
  Presentation: The Multi-Agent Approach: Building Reliable and Controllable Software Development Automation — read next: Read independent longitudinal/controlled measurement (METR's randomized study, full Faros methodology, DORA) and treat the vendor figures as directional, not established.
• uninformedThe central claim that 'the evidence is in' is asserted but the evidence itself is never presented — no measured outcomes, no specification of which patterns are effective or when. The Chatham House Rule and first-24-hours caveat partly explain this, but it leaves the confidence claim empirically ungrounded within the piece.
  Notes from FOSE Europe — read next: Seek empirical or DORA-style measurement studies of agentic-engineering outcomes, and the catalogued agentic-engineering pattern libraries the author mentions.
• incompleteThe post's own headline statistics (McKinsey 20-40%, Gartner 40%) are cited by source organization only, with no report named or linked, and the closing '80% on progress' claim about AI-leading companies has no citation at all — the piece's rhetorical force depends on numbers it doesn't let the reader verify.
  Thomas Nys' Post — read next: Trace the actual McKinsey and Gartner reports behind these figures to check whether '20-40% of tech estate value' and '40% of infrastructure systems' are being accurately summarized or rounded into a more alarming framing than the source data supports.
• incompleteThe 'receipts' section is careful to flag that other companies' headline migration numbers are vendor-reported or otherwise uncontrolled — yet the article's own prescriptions (zones, characterization-test-first, complete units, parallelize-last) are offered without any comparably rigorous measurement of whether following them actually produces better outcomes than not following them. The piece asks for more rigor from others' numbers than it applies to its own recommendations.
  Brownfield Agentic Engineering — read next: Look for controlled or telemetry-based studies (e.g. DORA, Faros) that test whether zone-based governance and complete-unit migration actually reduce defect/incident rates versus less structured approaches.
• uninformedThe methodology section discloses only respondent count (600, 530 completed) and survey window, with no detail on sampling frame, recruitment channel, or how 'stable layoff rates' compares to independent labor-market data.
  Engineering Leadership Report 2026 — read next: Cross-check the 'stable layoff rates' and burnout claims against independent sources — DORA's own 2026 report, the Stack Overflow Developer Survey, and labor-market tracking (e.g. layoffs.fyi) — to see whether these findings replicate outside a LeadDev/Postman-recruited leadership audience.
• incompleteThe finding that employee feedback (63%) dominates AI-impact measurement while rigorous metrics remain rare (~a fifth of orgs) is presented as a known weakness, but the report collects no data on why — cost, integration difficulty, lack of baseline data, or organizational incentive to avoid rigorous measurement are never explored.
  Engineering Leadership Report 2026 — read next: Read alongside extract-dx-measuring-ai-impact (confounded cross-sectional analyses, amplification/augmentation framework) and extract-leadership-ai-assisted-engineering (average AI metrics hide large per-company variance) for a fuller account of why rigorous AI-impact measurement remains difficult even for organizations that want it.
• incompleteAll figures come from Cursor, which has a commercial incentive for its Composer model to appear cheapest and for caching to look like its moat; there is no independent verification and the cost comparison is a single-agent-request figure, not a token-to-token one.
  The Pulse: Interesting AI coding stats from Cursor — read next: Seek independent, cross-vendor AI-usage and cost studies to corroborate or challenge the caching, cost-share, and acceptance-rate claims before treating them as general truths.
• replication and ecological generalizationN=34 finance professionals on one 10-hour valuation task with GPT-4o. Whether within-speaker persistence and task-switching dominance generalize to software engineering, healthcare, legal, or shorter sub-hour tasks is open. The paper's portability claim (rebuild task decomposition and knowledge graph) is undemonstrated outside finance.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Pu et al. 2025 'Assistance or disruption?' on proactive programming assistants (cited ref. 26) — closest existing analog in software domain. Also Kazemitabaar et al. 2024 (ref. 61) on interactive task decomposition in data analysis.
• normative-direction assumption'Precision proactivity' assumes calibrated scope-bounded proactivity is normatively superior to indiscriminate proactivity. But CLT has a 'desirable difficulty' tradition (intentional friction supporting learning), and serendipitous unsolicited surfacing — exploration that helps users find what they wouldn't have queried — is a real category. The paper conflates 'precision' with 'good' without engaging the exploratory case.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: van Merriënboer & Sweller 2005 on germane load (cited ref. 125) — the paper's exclusion of germane load from its operationalization is itself a theoretical commitment worth examining.
• construct validity vs alternative load measuresThe paper relies entirely on transcript-derived computational proxies. NASA-TLX, dual-task secondary performance, physiological measures (fNIRS, pupillometry) are listed as future-work triangulation. The 3× EL:ICL ratio is a strong claim resting on whether the two composites have comparable measurement reliability.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Babaei et al. 2025 on NASA-TLX limitations (cited ref. 54); Hirshfield et al. 2024 on fNIRS for cognitive load in multiple resources (cited ref. 69).
• cost / latency / model-economics blind spotSubtask focus stacks, explicit scope elicitation, and navigable architectures all imply additional compute, latency, and interaction friction. The paper does not engage with vendor cost models, latency-vs-quality tradeoffs, or the operational realities of deploying precision-proactive systems.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Xiao & Yang 2025 'Streaming, fast and slow: cognitive load-aware streaming for efficient LLM serving' (cited ref. 154) — the only operational-economics paper cited.
• attention infrastructure absent (hidden-layer thread)The paper treats cognitive load as intra-dialogue. It does not engage with the broader attention environment users bring to the work (notifications, multitasking, fragmented focus — Bailey, Stolen Focus, Hyperfocus). Extraneous load in the conversation arrives on top of ambient extraneous load in the environment, and the paper's instruments cannot see the latter.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Mark, Gudith & Klocke 2008 'cost of interrupted work' (cited ref. 36) is used only for intra-dialogue framing — Chris Bailey's Hyperfocus, Stolen Focus, and the broader attention-infrastructure literature would be the right complement.
• skill acquisition under sustained AI use (Beane signature)The paper raises in §7.4.1 the long-term schema-acquisition concern: 'if the self-sustaining load patterns we document prevent novice users from constructing coherent task schemas during AI-assisted work, the long-term consequence may extend beyond diminished output quality to impaired skill development.' Proposes longitudinal study as future work but does not test. Given Beane is co-author and The Skill Code's central concern is preserving conditions for skill acquisition, this is the most productive incomplete line in the paper. The one-shot 10-hour design is structurally unable to address it.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Matthew Beane, The Skill Code; van Merriënboer & Sweller 2005 on cognitive load and complex learning (cited ref. 125).
• governance / policy framing latentMishkin's policy lens does not surface — proactivity is treated as a design problem, not a policy or organizational-adoption problem. The 'precision proactivity' implications could be read as soft regulatory guidance for high-stakes professional contexts but the paper does not develop this. The latent policy angle (should regulators require scope-elicitation in high-stakes professional AI?) is not posed.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Appel et al. 2026 Anthropic Economic Index report (cited ref. 52) for the policy framing Mishkin and co-authors have produced elsewhere.
• RLHF interpretation as mechanism vs analogyThe Discussion proposes that prompt→response spillover reflects RLHF optimizing for relevance and helpfulness. This is plausible but mechanistic only by analogy — the paper does not actually instrument the model or correlate with known RLHF training details. Newer post-training paradigms (Constitutional AI, RLAIF, weighted preference learning, reasoning models) may produce different mirroring behaviors.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Examine whether reasoning-trained models (o-series, Claude 3.7+ extended thinking, GPT-5 reasoning modes) exhibit different prompt-mirroring spillover under the same measurement framework. Stiennon et al. 2020 and Ouyang et al. 2022 (refs. 110-111) are the cited RLHF foundations.
• extension vs relabel of classical CLTThe paper claims to extend CLT by reframing extraneous load as dynamic and path-dependent. But classical CLT has dynamic formulations (effective intrinsic load varies with expertise via schema automation — Kalyuga et al. 2003 expertise reversal). The contribution is sharper if read as: importing CLT's static-design-feature framing into the dynamic conversational setting required computational instruments classical CLT didn't have, which then revealed the static framing was insufficient. Whether this is 'extension' or 'translation' is not foregrounded.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Sweller 1994 (ref. 6); Kalyuga et al. 2003 expertise reversal (ref. 124); van Merriënboer & Sweller 2005 (ref. 125).
• harness-engineering syntopical bridgeThe paper does not engage with the agent-harness literature (compaction, sub-agents, context anxiety) but its within-speaker persistence finding is precisely the empirical grounding that literature needs. Each compaction event in a harness can be read as an attempt to break the self-sustaining load trajectory documented here.
  Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Anthropic harness paper on long-running application development; Turing Post 5 patterns for long-running agents; Cloudflare compaction-loss anecdotes.
• incompleteThe essay never addresses estimation-skill drift over the course of the study — teams could get better (or worse) at estimating simply from practicing the protocol, independent of any AI effect, which would reintroduce exactly the kind of within-team confound the four 'snags' were designed to eliminate. It is a plausible fifth snag the piece doesn't name.
  Measuring AI's Impact on Delivery Speed — read next: Look for discussion of calibration/learning effects in controlled measurement studies (e.g. how the METR study Shore credits for the design handled repeated-measures learning effects), or a statistician's treatment of practice effects in within-subject trial design.
• incompleteThe essay doesn't address ethical or practical constraints on random assignment — work that can't be randomly denied AI assistance (or forced to use it) for safety, regulatory, or externally-committed-deadline reasons is left out of the protocol entirely.
  Measuring AI's Impact on Delivery Speed — read next: Look for adaptations of within-subject experimental designs to constrained real-world engineering environments (e.g. regulated industries, on-call/incident work) rather than the idealized 'virtual bag' assignment described here.
• incompleteThe essay asserts 'it doesn't take many samples' and that 'the exact number will depend on the number of participants, the amount of variance in your results, and how conclusive you want the results to be' (footnote 7), but gives no worked numbers, formula, or example power calculation — a real gap for a reader who wants benchmark rigor over reassurance, since the whole essay otherwise leads with concrete numbers (Star Citizen's 112 points, the METR nearly-50% overestimate, the 2x/1.33x example).
  Measuring AI's Impact on Delivery Speed — read next: Look for the underlying METR study's own sample-size/power methodology (the design's stated inspiration), or a follow-up post from Shore with a worked example.
• incompleteShore's remedy for Goodhart's Law ('measure the approach, learn from it, then stop measuring') is not reconciled with his own point that AI's pricing and capability landscape is actively shifting (subsidies ending, prices potentially dropping, models improving). If an approach's cost/benefit profile can change after the fact, 'stop measuring' risks leaving an organization blind to exactly the kind of change the essay warns about earlier in the same piece.
  Are You Getting Your Money's Worth from AI? — read next: Watch for whether later parts of the series (or Shore's broader writing) specify a cadence or trigger condition for re-measuring an approach, rather than a one-time 'learn and stop.'
• incompleteThe value-add-percentage recommendation rests on a single practitioner's (Shore's own) three-year, single-organization experience, with no independent replication, cross-organizational benchmark, or comparison against alternative cost-of-change indicators — the same reliance on unverified personal anecdote that part 1 explicitly warned readers against trusting elsewhere in this very series.
  Measuring AI's Unintended Consequences — read next: Watch whether part 4 or the CFO epilogue supplies cross-organizational data on value-add percentage or a comparable indicator; compare against GitClear's larger-sample maintainability telemetry in extract-code-maintainability-plummets-ai-coding-era.
• uninformedThe '~70% AI-augmented' and 'three sessions is optimal' figures are self-reported, uninstrumented estimates from a single high-context practitioner (a VP with deep, privileged access to GitHub's own Copilot tooling and the authority to insist on rigorous SDLC infrastructure) rather than measured or benchmarked data, and may not generalize to engineers with less resourcing or organizational authority.
  Should You Still Become a Software Engineer in 2026? GitHub VP — read next: Compare against this corpus's instrumented AI-impact measurement sources (e.g. DX/Pragmatic Engineer throughput studies) rather than taking a single practitioner's self-report as representative.
• illogicalThe China-dominance argument compresses 'supply chain understanding' into a near-sufficient condition for long-run dominance. Other variables (capital allocation efficiency, demographics, talent migration, capability of domestic models, export-control regimes, energy supply, civil-military fusion costs) are not weighed.
  Simon Wardley's Post — Industrialisation of ML: nine notes on what to expect — read next: Read against Dan Wang on Chinese tech policy, CSIS/RAND supply-chain analyses, and any sceptical takes on PRC industrial policy execution (e.g. semiconductor self-sufficiency progress vs targets).
• incompleteThe Jevons-paradox reframing (adjacent possibles becoming real) is asserted but the conditions under which adjacent possibles actually become real industries — vs stalling — are not specified. Sometimes the adjacent possible is real and stays small.
  Simon Wardley's Post — Industrialisation of ML: nine notes on what to expect — read next: Read against the slower-than-expected industrialisation histories — early electricity (the 40-year productivity paradox), enterprise software in the 1990s. Beck and Shoup on Jevons-in-coding is a sharper instantiation of this same claim worth juxtaposing.
• uninformed (mild)The post under-weights regulatory ratchets — particularly EU AI Act, sector-specific liability regimes, GDPR-style data sovereignty rules — as climatic variables that shape what 'industrialised ML' actually looks like in different jurisdictions. Wardley's mapping treats regulation as doctrine; this post treats it as background.
  Simon Wardley's Post — Industrialisation of ML: nine notes on what to expect — read next: Read alongside EU AI Act implementing acts, NSM/Datatilsynet guidance for the Norwegian context, and Frederiksen on Coop Bank sovereignty for how an enterprise actually reasons about regulatory ratchets as architectural constraint.
• incompleteThe post is silent on the cultural and organisational variation in how 'practice converges' over the 5-8 year window. Nordic public-sector institutions, Japanese manufacturers, and Silicon Valley startups will not converge to the same practice — but the post talks as if there is one convergence trajectory.
  Simon Wardley's Post — Industrialisation of ML: nine notes on what to expect — read next: Read against Cutler on organisational operating systems and the Heis stable's Norwegian-leader interviews — concrete evidence that practice-convergence is mediated by national culture and institutional shape, not just by elapsed years.
• incompleteNo evidence is offered that the workflow improves outcomes. Every recommendation rests on the author's personal, weeks-long use ('outrageously useful', 'it brings me joy') and on adoption metrics (160K stars, 7M downloads on skills.sh) that measure popularity, not code quality, delivery speed, or defect rate. There is no controlled comparison against ad-hoc prompting or alternative flows, and no account of the added review burden that a mandatory code-review stage over many agent-session tickets creates.
  New Skills! v1.1 brings /wayfinder, /research, /implement, /to-spec, /to-tickets — read next: Controlled studies on whether staged / spec-driven agent flows beat ad-hoc prompting; DORA-style measurement of agentic-engineering outcomes; work on whether the review burden from many agent-produced tickets becomes the binding constraint.
• incompleteThe central benefit — reduced brittle-test maintenance and improved resilience — is asserted with zero quantification. There is no flake-rate before/after, no measure of how often the agent successfully adapts versus how often it produces a wrong or non-reproducible result, and no comparison of maintenance effort against scripted tests.
  Slack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation — read next: Read the linked Slack engineering post directly for any concrete metrics; compare with evaluation work that measures agentic vs scripted E2E flake and maintenance rates. Absent numbers, treat the resilience claim as a hypothesis.
• incompleteVendor incentive distortion in the 'whole ecosystem is accelerating' claim
  Itchy Brain (Still Burning: Michael Grinich & Kent Beck on AI adoption, enterprise developer tools) — read next: DORA 2025 and the Stanford/Microsoft developer-productivity studies on AI's actual code-quality effects; Gergely Orosz on enterprise sales-cycle compression; Marc Andreessen's classic 'why software is eating the world' for the prior pattern. For the contrarian counterweight: Gary Marcus and Ed Zitron on whether the AI productivity story is being narrated by the people selling it.
• incompleteThe marginal-cost claim is accounting-incomplete (Beck flags it; Grinich doesn't return)
  Itchy Brain (Still Burning: Michael Grinich & Kent Beck on AI adoption, enterprise developer tools) — read next: Anything in your Kent Beck file on TCR and scope-of-done (this is exactly the conversation about plausible deniability and shrinking scope); Lorin Hochstein on operational complexity; the Böckeler harness-engineering extraction in this repo (the 'gap between running system and trusted system' is the harness-engineering surface). The Adam Tooze / Brad DeLong line on 'productivity gains that don't show up in the statistics' is the macroeconomic complement.
• incompleteThe 'descaling of software' counter-trend is named by Beck and left hanging
  Itchy Brain (Still Burning: Michael Grinich & Kent Beck on AI adoption, enterprise developer tools) — read next: Steve Yegge on vibe coding; Gene Kim's vibe coding series; Simon Willison on the 'no-coder using LLMs to build real software' phenomenon; the Schoening agency-over-skills extraction in this repo (the agency thesis is the descaling-friendly side of this tug of war). Honest framing: Beck's craft instincts and Grinich's institutional instincts pull in opposite directions; you may have to choose.
• incompleteGrinich's 'industrialization phase' analogy launders externalities
  Itchy Brain (Still Burning: Michael Grinich & Kent Beck on AI adoption, enterprise developer tools) — read next: Tooze on energy transitions; the IEA's compute-electricity projections; the literature on whether the late-19th-century industrialization actually 'cleaned up' or just exported externalities. For Kjetil specifically: this is the systems-thinking lens — feedback loops, stocks/flows, who absorbs what — applied to the rhetoric of inevitability.
• incompleteWhat changes about Beck specifically — the craft thinker in this regime
  Itchy Brain (Still Burning: Michael Grinich & Kent Beck on AI adoption, enterprise developer tools) — read next: Beck's own recent Substack posts (Tidy First, the 'starving genies' piece, 'forest thinning'); your highlights on Beck for where he reframes craft for the agentic era; Hickey's 'Simple Made Easy' as the substrate craft argument; Ousterhout's complexity work. The persona-relevant cross-thread: this is where Kjetil's 'fragility of ideas' hidden layer touches the AI transition — if the craft of attentive code-reading is the protected attention surface, what protects it now?
• incompleteWhat does the 'itchy brain' protect against, exactly?
  Itchy Brain (Still Burning: Michael Grinich & Kent Beck on AI adoption, enterprise developer tools) — read next: Pressfield on Resistance; Cal Newport on deep work degradation under tool proliferation; the David Whyte/Maria Popova line on the difference between making and consuming; the persona's Bailey/Hyperfocus annotations on what attention needs in order to do anything. This is where the persona's hidden layer (fragility of ideas) and the episode's title concept (itchy brain) collide — and it is the most productive question the episode opens.
• uninformedThe central quantitative claim (30-100x productivity) and the five-year prediction (no AI-naive Scrum team employable by 2030) are stated with no methodology, data source, or citation. This is the kind of confident, unqualified AI-transformation prediction that other documents in this corpus explicitly warn against (practitioners elsewhere in the corpus argue nobody has the data to be that confident) and that measured studies (DORA) show producing far smaller, sometimes negative, effects.
  Is Agile Dead? How AI and Extreme Agile Redefine Work — read next: Compare against DORA's 2024/2025 State of DevOps findings, which measure AI-adoption effects in low single-digit percentages, not multiples — a direct empirical counterweight to this article's order-of-magnitude claim.
• incompleteThe newsletter juxtaposes the OpenClaw story (passionate builder using AI agents productively) with the AI Psychosis critique (executives using AI agents unproductively) but never articulates what distinguishes productive from unproductive AI agent use. The implicit answer seems to be 'building vs. managing' but this is never made explicit or tested.
  SWLW #702: How I created OpenClaw, Your CEO is suffering from AI Psychosis, and more. — read next: Read the actual OpenClaw video/article and the AI Psychosis article side by side to extract the distinguishing criteria
• incompleteThe compounding argument (10% effort → 2x output) is stated as fact but has no empirical backing cited. The Hamming lecture is anecdotal authority, not evidence. The actual compounding dynamics of knowledge work are far more complex than financial interest.
  SWLW #702: How I created OpenClaw, Your CEO is suffering from AI Psychosis, and more. — read next: Cal Newport's 'Deep Work' or Ericsson's deliberate practice research for more rigorous treatments of effort-to-outcome dynamics
• incompleteThe curator's editorial voice synthesizes across articles but never interrogates the tension between 'seek measurable value' (AI Psychosis commentary) and 'shape what's possible rather than optimize what's in front of you' (strategic thinking commentary). These are potentially contradictory: strategic thinking often requires investing in things that are not yet measurable.
  SWLW #702: How I created OpenClaw, Your CEO is suffering from AI Psychosis, and more. — read next: Roger Martin's work on strategy as an integrative activity, or Clayton Christensen on the innovator's dilemma of measurability bias
• incompleteThe issue diagnoses AI-era pressures (salary-scale economics, direction-less amplification, agent reliability, term-inflation) and poses them as open questions to the reader ('where does your company sit?', 'what would you do differently?') but never synthesizes them into an operating model — it stops short of what a leader should actually do.
  SWLW #710: Revised rules of Eng Leadership, When AI costs more than the engineer, and more. — read next: Will Larson's 'Revised Rules of Engineering Leadership' itself (the linked favorite), and John Cutler / engineering-strategy sources that convert principles into concrete decision rules.
• incompleteThe continuity framing (fundamentals and judgment are what matter, AI only raises their stakes) under-weights the possibility that some fundamentals themselves must change under AI — it treats 'fundamentals' as fixed rather than a live question.
  SWLW #710: Revised rules of Eng Leadership, When AI costs more than the engineer, and more. — read next: Sources debating whether the fundamentals taught today are still the right ones (e.g. Tobias Torrissen on juniors and fundamentals; 'The Future of Software Engineering' on mid-level retraining).
• incompleteNone of the four variables is defined operationally or made measurable -- there is no way to observe, score, or compare 'Machine Understanding,' 'Problem Understanding,' 'Practice Evolution,' or 'Social Contract' for a given organization, so the essay's most emphatic claim (denominator collapse dominates any numerator strength) cannot actually be tested against a specific case.
  TBM 431: The Denominator That Matters — read next: TBM 430's own unresolved gap about un-operationalized axes (extract-tbm-430-incubate-compound-refinance-liquidate#g1); DORA/SPACE literature on operationalizing soft organizational constructs into measurable proxies.
• illogicalThe formula's multiplicative structure and 'denominator' framing borrow the rhetorical authority of arithmetic (multiplication, division, zero-collapse) to argue a qualitative organizational claim, but none of the four terms are actual quantities -- there is no unit, no scale, and no independently verifiable zero point for any of them. Treating 'the formula is multiplicative' as a premise that licenses 'the denominator is not optional' smuggles more logical necessity into the argument than a metaphor can bear.
  TBM 431: The Denominator That Matters — read next: Compare with other multiplicative-formula rhetoric in the corpus (e.g. TBM 430/432's 'physics vs. variables' framing, which stays at the level of causal structure rather than literal arithmetic) to see whether Cutler's other pieces make the same move more cautiously.
• incompleteThe essay names six failure-mode combinations for the three numerator variables but treats each as strictly present-or-absent; it never explores degrees -- what happens when a term is weak rather than zero (shallow Machine Understanding, superficial Problem Understanding)? The all-or-nothing 'zero kills the product' framing may not describe real organizations, where these variables are usually partial and blended rather than binary.
  TBM 431: The Denominator That Matters — read next: Compare with TBM 432's more granular nine-motion taxonomy of capability change (extract-tbm-432-bundling-unbundling-capabilities), which at least allows partial/simultaneous states rather than binary presence/absence.
• incompleteCutler never specifies who is supposed to build, own, and adjudicate disputes about the 'theory of the system' / causal model he proposes -- finance, engineering leadership, product, or some new function. Without a named owner and a process for contesting a causal narrative's validity, 'make your theory of the system explicit' risks becoming just as rhetorically gameable as the proxies it replaces: a leader can write a confident-sounding, well-formatted causal story without it being falsifiable in practice, and nothing in the piece specifies who would catch that.
  TBM 437: Tokens, Hours, Points, and Other Curious Proxies — read next: Organizational-design and FinOps-style AI-cost-governance literature on who owns cross-functional investment decisions; DORA's Capabilities Model research on how measurement gets operationalized organizationally rather than just prescribed.
• incompleteThe experiment is a single case: one codebase, one change type (adding a trait with three methods), one model/harness (Claude Code / Sonnet 5), one developer, no repeated trials per step to establish variance. This can't yet distinguish 'refactoring reliably reduces agent token cost' from 'this particular refactoring sequence, on this particular file, happened to.'
  The Economic Benefit of Refactoring — read next: Look for replications across different codebases, languages, and change types. Compare against extract-almedra-token-efficiency-benchmark-glm-5-2, which faces a structurally identical single-task/single-repo generalizability gap from the opposite angle (harness choice rather than codebase refactoring) — the two pieces would benefit from being read together as parallel single-case token-efficiency studies in need of replication.
• uninformedThe document is an industry-wide, cross-client synthesis produced by a 22-person advisory board; it does not — and structurally cannot — speak to how any of these recommendations play out inside a specific team's existing culture, legacy codebase, or organizational politics. Every blip is stated as a general recommendation, not a contingent one.
  Technology Radar, Volume 34 — read next: Pair with team-level case studies or first-person practitioner accounts (e.g. this corpus's own harness-engineering and agent-config-decay extractions) to ground the Radar's generalizations in a specific team's experience.
• incompleteThis is a single practitioner's (the author's own) account, with no data on team-scale adoption: whether the 'tests as human/agent boundary' approach holds up across a team of varying skill levels, or whether it reproduces the review-bottleneck and quality costs that corpus-level telemetry (Faros, DORA) documents for larger, less-granular AI-authored changes.
  Practices I Abandoned with Agents: An Ode to Test-Driven Development — read next: Cross-check against org-scale telemetry on PR size, review time, and defect rates under AI-assisted development to see whether 'go bigger, verify at the boundary' generalizes past a founder's own codebase.
• incomplete'Net code quality ratio' and 'compute efficiency per feature' are proposed but never shown implemented or validated with real numbers — no worked example of how post-release hotfixes get attributed back to AI-authored versus human-authored code in mixed-authorship commits.
  Uber's Zero Growth Stack: Scaling Services, While Optimising Infrastructure and AI Cost — read next: Follow-up Uber engineering posts or corpus material on turn-level agent cost attribution and AI-impact measurement methodology, to see whether such attribution has actually been operationalised anywhere.
• incompleteThe optimistic evidence is uncontrolled, single-team, self-reported metrics (a sample of one), which the author admits carries no conclusions; the piece names the asymmetry but does not offer a path to controlled, comparable evidence for the upside.
  What Survived Contact — read next: Look for controlled or quasi-experimental studies of agentic-coding productivity and quality, and for methods that make single-team metrics comparable across teams.
• incompleteHe asserts taste cannot be tested but never addresses how taste is developed or transmitted. He brings pre-existing taste to every judgment (the population-map problem, the anglophone-Wikipedia bias, the suspect Reykjavik=10) yet treats it as a given human endowment — silent on how a novice, a team, or the next generation would acquire or inherit it, especially once AI removes the on-ramp tasks that used to build it.
  You can't unit test for taste — read next: Read Polanyi on tacit knowledge and Dreyfus on skill acquisition (novice-to-expert); pair with the corpus's junior-apprenticeship thread (heis-09, heis-12, ki-utvikling-og-folelser) and Beck 'Tidy First?' / Ousterhout on design judgment as cultivated rather than innate. Ask: does taste transmit at all when AI removes the practice that used to form it?
• incompleteHe confines 'you can't unit test for taste' to messy data and treats code as the safely 'verifiable domain,' but never turns the lens onto AI-generated code itself, where tests pass yet design taste — simplicity, deep modules, restraint — has no red/green check. As agents generate more test-passing code, the untestable-taste problem migrates into the codebase, not away from it.
  You can't unit test for taste — read next: Read Ousterhout 'A Philosophy of Software Design,' Beck 'Tidy First?,' and Hickey 'Simple Made Easy' on design quality as unassertable; cross with corpus docs agents-dont-know-what-good-looks-like (behavioral vs capability verification) and cognitive-surrender (surface-correctness signals). Test the claim: is a green build ever evidence of good design?
• incompleteHe delegates the 'subjective' rating to the LLM's latent weights and finds it useful, but never examines whether that latent 'taste' is genuine judgment or averaged internet popularity — i.e., whether outsourcing scoring to a model launders the very anglophone-Wikipedia bias he diagnosed a section earlier into a 'subjective score' he can no longer see or audit.
  You can't unit test for taste — read next: Read on models as reflections of their training distribution (Bender et al.) and on popularity-as-proxy-for-quality critiques; probe the LLM ratings against non-anglophone sources to see whether the 'taste' is judgment or a re-encoding of who edits Wikipedia.