Case Study: Refactoring at Scale with Agents

codescene.com · article · 2026-09-24 source ↗ Reader 🖥️

practical · narrative case study with argumentative framing — Written by Adam Tornhill (CodeScene founder/CTO) as a vendor case study: narrates a concrete large-scale refactoring project (credited to Daniel Webb and Dr. Markus Borg) and argues from it to a general claim about the economics of technical-debt remediation.

Structure

Key terms

• Code Health Behavioral Code Analysis
  CodeScene's proprietary 0-10 code-quality score, used here as the deterministic target metric that refactoring agents optimize against and are judged by — chosen specifically because it was independently validated (in prior CodeScene research) as a proxy for productivity, defect density, AI correctness, and token efficiency, not merely a convenient automatable number.
• Playbook
  An iteratively-built, versioned set of named refactoring recipes (with documented preconditions, success/failure outcomes, and additional notes) that the agents accumulate as they work through a codebase, making each subsequent refactoring task more effective than the last.
• Domain-specific refactoring recipe (e.g. 'Shared Index Range', 'Action Parameter', 'Uniform Step Table')
  A named code transformation discovered by the agents that is specific to patterns in this particular codebase, structurally different from Martin Fowler's generic refactoring catalogue (Extract Function, Guard Clauses, Parameter Object, etc.), which the author treats as the generic 'foundation' that domain-specific recipes then adapt.
• AI-ready code
  Code structured such that AI coding agents perform reliably and cost-efficiently on it — the stated end goal of the refactoring, and the reframing of 'code quality' as something done in service of AI performance, not only human maintainability.

Propositions

• explicitcorroboratedPoorly structured code and technical debt are a direct, measurable barrier to agentic coding performance, not only a human-maintainability concern.conf: high
  supports ←Brownfield Agentic Engineering · p1 — Both treat poor codebase structure/technical debt as a direct, measurable constraint on agent performance, not just a human-maintainability concern — motivating the zone/characterization-test discipline here and the Code-Health-first discipline there.
• explicitA 300,000-line C codebase (Street Fighter III: 3rd Strike) was refactored from Code Health 5.6 to a perfect 10.0 in three weeks at roughly $4,000 in token cost, versus an estimated 12-18 months of expert developer time in the pre-AI era — an order-of-magnitude reduction in elapsed project time and cost.conf: high
  supportsIs AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard · p10 — Empirically demonstrates, at real scale with real numbers, that legacy modernization is a tractable, worthwhile target — corroborating that it's a genuine bottleneck-aligned use case rather than an aspirational one.
  extendsForskning på digitalisering 2026 uke 19 · p9 — Where that document asserts LLMs 'can likely' refactor legacy code faster/better than humans, this case study supplies a concrete, measured instance (300k LOC, 3 weeks, $4,000) with explicit qualification of failure modes.
• explicitcorroboratedFunctional correctness throughout the refactor was verified by a replay-trace harness comparing rollback-state hashes frame by frame, not by human code review alone — automated, deterministic equivalence checking was the actual safety mechanism that made large-scale agentic refactoring trustworthy.conf: high
  extendsThis coding agent doesn’t want your feedback — it ships without it · p6 — Both describe a closed-loop agent pattern (deterministic feedback, bounded autonomous iteration), but this case study demonstrates it at far larger scale and with a richer verification oracle (frame-exact replay-trace equivalence) than a type-checker-and-retry loop.
  supports ←Brownfield Agentic Engineering · p8 — Both use a shadow/replay-comparison oracle (diff old vs. new behavior on real traffic or recorded state) as the substitute for a trustworthy test suite — the same pattern applied to a web backend here and to a game-engine refactor there.
• explicitA larger, more capable model (Claude Opus) substantially outperformed smaller models (Sonnet, 'Terra'; also compared against Codex with 'Sol') on this task specifically because of its ability to capture and document discovered refactoring patterns into the emerging playbook, not just raw code-generation quality — smaller models plateaued at a local optimum on individual files and could not progress on remaining code smells.conf: high
• explicitThe agents discovered and named domain-specific refactoring recipes (e.g. 'Shared Index Range', 'Action Parameter', 'Uniform Step Table') that are structurally different from the generic refactoring catalogue (Fowler) a human expert would apply, suggesting generic refactorings are a foundation on which domain-specific, AI-discovered refactorings then build.conf: high
• explicitNot all refactoring attempts succeeded — some made Code Health worse — and documenting these failures in the playbook alongside successes was treated as essential to the process's effectiveness, not an embarrassing footnote.conf: high
• explicitcorroboratedCode Health was trusted as the agent's optimization target specifically because it had already been independently validated (in prior CodeScene research) as a proxy for real engineering outcomes — 10x faster evolution and 15x fewer defects in healthy code, improved AI correctness, and reduced token usage — which the author explicitly frames as the answer to the risk that 'give an agent a metric and it will happily optimize for any metric.'conf: high
  supportsFragments: February 18 · p12 — Corroborates, from a different research lineage (CodeScene's AI-Ready-Code whitepaper cited in this article), the claim that LLMs behave more reliably on healthier codebases — the whole case study's premise is to engineer that healthiness first.
  supports ←Code Red: The Business Impact of Code Quality -- A Quantitative Study of 39 Proprietary Production Codebases · p1 — This paper is the original quantitative study (Tornhill & Borg, 2022) behind the '15x fewer defects' figure that the later agentic-refactoring case study cites as the prior independent validation justifying trust in Code Health as an agent's optimization target.
  extends ←Practices I Abandoned with Agents: An Ode to Test-Driven Development · p6 — Same author/company: Code Health is trusted as an agent optimization target precisely because it's an independently-validated, tooling-enforced proxy — the same 'enforce via tooling, not manual inspection' move this article makes for tests.
• explicitcorroboratedTechnical debt is 'a more urgent problem than ever, not less' after this case study, because organizations that adopt AI coding naively onto unhealthy codebases are likely to compound existing problems rather than see productivity gains materialize automatically.conf: high
  supports ←Thomas Nys' Post · p4 — Both argue technical debt becomes more, not less, urgent once AI coding is adopted, because naive AI adoption onto unhealthy codebases compounds rather than resolves the underlying problem.
• implicitThe case study's favorable economics may not generalize to a typical enterprise legacy system: the codebase was an open-source game decompilation with deterministic, frame-perfect replayable state (enabling the correctness harness) and no external dependencies, databases, concurrency, or production constraints — conditions most enterprise codebases don't share.conf: medium

Arguments

Narratives

• The Street Fighter III: 3rd Strike case study itself: 300,000 lines of 1990s arcade-game C code, refactored under a replay-trace verification harness that confirmed 'we could still play the game as before.'
  → Makes the correctness-verification claim viscerally concrete and credible — a deterministic, replayable game state is a case where 'did we break anything' has an unusually clean, automatable answer, which is exactly what makes the rest of the argument's safety claims land.
• A quoted practitioner reaction: 'I had my agentic coding wins in the past, but this is the first time I'm experiencing a true superhuman AI performance at scale. And I say that as someone with three decades of experience in large-scale system development.'
  → Supplies a credibility appeal from a named, experienced practitioner (implicitly Daniel Webb or Markus Borg) rather than the vendor's own marketing voice, supporting the magnitude claim about model capability (proposition 4).

Questions addressed

• ✅ Can coding agents remediate large-scale technical debt economically, in a domain where human-driven rewrites have historically failed or been deprioritized as too risky and expensive?
  Yes, demonstrated at 300k LOC for roughly $4,000 and three weeks, given a deterministic quality-feedback loop and an automated correctness-verification harness.
• ✅ Which refactorings should an agent apply at scale, and are they the same ones a human expert would reach for?
  No — the agents discovered and productized domain-specific recipes structurally different from a human's generic catalogue (Fowler), which the author treats as a genuine, if 'obvious in retrospect,' insight: generic refactorings are the foundation, domain-specific ones the adaptation.
• ✅ Does this result mean technical debt is now a solved problem industry-wide?
  No — explicitly addressed and rejected: naive AI adoption onto unhealthy codebases, without this feedback/verification infrastructure, is argued to compound debt rather than resolve it.
• ✅ Is the Code Health metric a legitimate optimization target for an agent, or just a convenient, gameable proxy?
  Argued legitimate, because it was independently validated against productivity, defect-density, AI-correctness, and token-efficiency outcomes before being adopted here — directly answering the Goodhart's-law risk the author names himself.

Gaps

• incompleteThis is an n=1 case study, authored by the vendor of the Code Health tool being validated, on a codebase chosen specifically for having deterministic, frame-perfect replayable state — a correctness-verification advantage most enterprise codebases (external APIs, databases, UI, concurrency, production constraints) don't have. The article never addresses how correctness would be verified at this pace and scale without an equivalent deterministic oracle.
  Read next: Look for a second, independent (ideally non-vendor-authored) case study on a typical enterprise codebase lacking a replay-trace-style oracle, to test whether the result generalizes.
• incompleteThe headline $4,000-tokens-vs-12-18-months-of-engineers comparison discloses neither the loaded engineer cost assumption, the token pricing used, nor whether the 12-18 month estimate is the author's own guess or drawn from a cited prior project — leaving the ROI figure directionally persuasive but not independently checkable.
  Read next: Request or look for the underlying cost model and a citation for the 12-18 month baseline estimate.
• incompleteThe article states that 'many' refactoring attempts failed or made Code Health worse, and that documenting these failures mattered, but gives no quantitative failure rate — e.g. how many attempts were needed per successful recipe, or what fraction of the 22 recipes required multiple tries. This number would materially change how repeatable (versus lucky) the three-week headline figure looks.
  Read next: A follow-up post or dataset disclosing attempt-level success/failure counts, not just the final playbook.
• incompleteThe playbook and its named recipes were discovered on a single codebase (C, a 2D fighting-game engine). The article doesn't address how much of the discovered playbook is reusable across codebases/domains versus specific to this one — which bears directly on whether the $4,000/three-week economics would hold on a second, unrelated codebase rather than being a one-off artifact of this particular code's structure.
  Read next: A second case study on a different language/domain (e.g. a typical enterprise CRUD service) testing playbook transfer.

Closes gaps elsewhere

• Forskning på digitalisering 2026 uke 19 · g2 — Supplies exactly the evidence and failure-mode qualification that gap asked for: a measured before/after Code Health transformation with cost and time figures, a model comparison showing weaker models plateau at local optima, and explicit acknowledgment that many refactoring attempts failed or degraded quality along the way.

Related documents

• Fragments: February 18
• Forskning på digitalisering 2026 uke 19
• Code maintainability plummets in the AI coding era
  Code Quality Legacy & Organizational Forgetting Technical Debt
• Coding Models Are Doing Too Much
  Code Quality Coding Agents Legacy & Organizational Forgetting
• The Economic Benefit of Refactoring
  Code Quality Technical Debt Token Efficiency
• Code Red: The Business Impact of Code Quality -- A Quantitative Study of 39 Proprietary Production Codebases
  Behavioral Code Analysis Code Quality Technical Debt
• Why did Martin Fowler invite 60 engineers to a ski resort?
  Code Quality Coding Agents Legacy & Organizational Forgetting
• Agentic Engine Optimization (AEO)
  Coding Agents Token Efficiency
• [FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero
  Coding Agents Technical Debt
• The Almedra Token-Efficiency Benchmark, Updated For GLM 5.2
  Coding Agents Token Efficiency