Code Red: The Business Impact of Code Quality -- A Quantitative Study of 39 Proprietary Production Codebases

Adam Tornhill, Markus Borg · pdf · 2022-03-08 Reader 🖥️

AI-Friendly Code (Code Health for Agents)Behavioral Code Analysis Code Quality DORA & Engineering Metrics Technical Debt

Code quality, measured via CodeScene's Code Health metric, has a quantifiable and substantial business impact — low-quality code produces 15 times more reported defects, takes 124% longer to resolve on average, and is up to 9 times less predictable to resolve than high-quality code — so code quality must be treated as a business concern tracked with the same rigor as financial KPIs, not dismissed as a purely technical aesthetic.

theoretical · argumentative — An empirical research paper (repository-mining study across 39 proprietary codebases) building toward a quantified causal-style claim about code quality's business impact, in explicit service of a practical conclusion: treat code quality as a tracked business KPI, not an abstract technical nicety.

Structure

Part	Role
Introduction: motivates the problem via the developer shortage, hyperbolic discounting of code quality, and the absence of code-quality KPIs that are legible to business managers	context
Background: defines Code Health (CodeScene's 1.0-10.0 proprietary metric, validated against lines-of-code as a weaker predictor) and Time-in-Development (a file-level cycle-time metric derived by combining Jira status transitions with commit metadata)	method/definitions
Related work: positions the study as complementary to DORA's Four Key Metrics (which measure delivery/ops, not the coding stage itself) and to prior technical-debt-interest research	context
Method: repository mining across 39 proprietary CodeScene-customer codebases (30,737 files, 14 languages, 6-12 months of history each), three research questions (defect correlation, dev-time cost, predictability)	evidence/method
Results: Kruskal-Wallis and ANOVA tests showing statistically significant differences across Code Health categories for defect counts (RQ1), average Time-in-Development (RQ2), and maximum Time-in-Development (RQ3)	evidence
Discussion: reframes the numbers in business terms (time-to-market, cost of defects, predictability-driven stress) and argues code quality is a sustainability precondition for the industry	conclusion
Threats to validity and future work: acknowledges causality is unestablished, sample is CodeScene-customer-only, and proposes studying author-experience effects and per-language differences	qualification

Key terms

Code Health Behavioral Code Analysis
CodeScene's proprietary 1.0 (unhealthy) to 10.0 (healthy) score, combining ISO 5055 maintainability elements with design-level code smells (God Class, God Method, Duplicated Code); files are bucketed as Healthy (>=8.0), Warning, or Alert (<4.0).
Time-in-Development
A file-level cycle-time metric the authors construct themselves: the accumulated time a file was under active work for a given Jira issue, derived by combining Jira 'In Progress' transition timestamps with the timestamps of commits referencing that issue (summed across sub-cycles when an issue spans multiple commits).
Technical Debt interest
Following the TD community, the authors distinguish the debt principal (the remediation/refactoring cost) from the interest (the recurring additional development time paid on every future change touching the debt-laden code) — the paper measures interest, not principal.
Hyperbolic discounting
A decision-making bias (borrowed from behavioral economics, Laibson 1997) applied to explain why organizations repeatedly choose short-term time-to-market/features over code quality, even though their future selves bear higher costs for that choice.

Propositions

explicitLow-quality code (Alert Code Health category) contains on average 15 times more reported Jira defects than high-quality code (Healthy category).conf: high
supportsCase Study: Refactoring at Scale with Agents · p7 — This paper is the original quantitative study (Tornhill & Borg, 2022) behind the '15x fewer defects' figure that the later agentic-refactoring case study cites as the prior independent validation justifying trust in Code Health as an agent's optimization target.
explicitResolving an issue in low-quality (Alert) code takes on average 124% longer than resolving a comparable issue in high-quality (Healthy) code; even moderate-quality (Warning) code takes 78% longer than Healthy code.conf: high
explicitLow-quality (Alert) code is up to 9 times less predictable to resolve than high-quality code, measured as the maximum Time-in-Development for an issue; Warning code already shows more than double the maximum Time-in-Development of Healthy code.conf: high
supportsThe Economic Benefit of Refactoring · p9 — This paper establishes Code Health's predictive validity for human-era defect rates and dev-time variance; the later token-cost experiment extends the same 'code quality as a leading-indicator proxy' logic into agentic token economics.
explicitCode Health correlates with issue resolution time far more strongly than a naive lines-of-code metric (Pearson r=-0.58 for Code Health vs. r=0.13 for LoC), establishing Code Health as adding predictive value beyond simple size.conf: high
explicitOrganizations systematically trade code quality for time-to-market and new features because of hyperbolic discounting, and because code quality's business impact has historically lacked a finance-comparable, quantifiable KPI.conf: high
explicitThe lack of standard KPIs for code quality explains why, in a study of 15 large organizations, business managers often would not grant the budget or priority needed for refactoring even when developers flagged technical debt.conf: high
explicitThe cost of technical debt is not the remediation work itself but the recurring 'interest' — the additional development time paid on every future change that touches debt-laden code.conf: high
extendsSoftware Design X-Rays · p1 — Both papers share the TD-interest framing (debt is the recurring interest payment, not the remediation cost); this paper supplies the quantified interest rate (124% / 9x) that the book's hotspot/interest-rate metaphor leaves qualitative.
explicitThis study is positioned as complementary to DORA's Four Key Metrics: DORA measures delivery/ops performance (lead time for change, deployment frequency, time to restore, change failure rate), while this study measures the earlier stage — the waste introduced when code is actually written.conf: high
explicitLow predictability in resolving issues (high variance in Time-in-Development) causes organizational stress, since unpredictable task completion puts commitments and deadlines at risk for managers.conf: medium
explicitAs few as 7.2% of organizations methodically track technical debt, despite TD being estimated to waste up to 42% of developer time -- awareness of the problem is decoupled from measurement and action on it.conf: medium
implicitDecision-makers will change code-quality-related budgeting and prioritization behavior once given quantified, finance-comparable numbers for the cost of low quality.conf: medium
implicitA developer's degree of ownership/experience with a specific low-quality file plausibly reduces the variance and duration of resolving issues in that file, though this is flagged as a hypothesis for future work rather than a finding of this study.conf: low

Arguments

Code quality must be treated as a fundamental business goal, tracked and prioritized with the rigor of a financial KPI. strength: medium

Explicit premises:

Technical debt is estimated to waste up to 42% of developer time, costing an estimated $85 billion annually.
No standard, finance-legible KPI for code quality currently exists, which is why managers don't grant budget for refactoring.
This study's data (39 codebases, 30,737 files) shows Code Health statistically significantly predicts defect counts, development time, and predictability of development time.

Implicit premises:

Decision-makers currently withhold investment in code quality primarily because of a measurement/legibility gap, not because of competing priorities, incentive structures, or genuine uncertainty about returns.
A vendor-produced, proprietary metric (Code Health) is an acceptable and sufficiently neutral basis for an industry-wide call to action, despite being owned by one of the paper's own authors' companies.

Inferences:

If code quality this reliably predicts defect rates and delivery time, then software effort/time estimation practices that ignore code quality are necessarily miscalibrated, and quality should be a required input to any credible estimate.
Since the Warning category already shows a meaningful cost (78% longer dev time), organizations should treat 'Warning' as actionable, not just 'Alert' — the paper's own framing understates this by headlining only the Alert-vs-Healthy contrast.

Questions addressed

✅ RQ1: How does the number of reported defects in source code files correlate with Code Health?
Alert-category files have ~15x more reported Jira defects on average than Healthy-category files, a statistically significant difference (medium-to-large effect size between Healthy and Alert).
✅ RQ2: How much longer does development take to resolve an issue in low-quality source code?
On average 124% longer in Alert code and 78% longer in Warning code, compared to Healthy code.
✅ RQ3: Is resolving issues in low-quality code less predictable (higher risk of schedule overrun)?
Yes -- maximum Time-in-Development is up to 9x higher in Alert code, a large effect size, indicating substantially higher variance/schedule risk.
⭕ Does low code quality cause more defects and slower, less predictable development, or do defects/time pressure cause low code quality (reverse causality)?
The paper explicitly flags this as an open internal-validity threat and proposes randomized controlled trials or Bayesian causal analysis as future work, but ships its headline business conclusions in strongly causal-sounding language in the meantime.

Gaps

incompleteThe study's data and conclusions are entirely pre-coding-agent (mined circa 2021, published March 2022): it establishes that Code Health predicts human-developer defect rates and cycle time, but says nothing about whether the same metric predicts AI-agent performance on the same code. That extension is only tested elsewhere in this corpus, not in this paper.
Read next: See extract-codescene-refactoring-at-scale-with-agents for the agentic-era follow-on that treats this paper's validated metric as the optimization target for a coding agent refactoring a 300k-line codebase.
illogicalCausality between Code Health and the dependent variables (defects, dev time) is not established, only correlation across 39 codebases -- the paper itself names this as a threat to internal validity (low quality could be a symptom of time pressure or churn rather than a cause) -- yet the Discussion and Conclusion sections repeatedly use causal framing ('the business advantage of high quality code should be unmistakably clear') without hedging to the same degree as the Threats to Validity section.
Read next: Look for a follow-up randomized or quasi-experimental study (the authors propose RCTs or Bayesian causal analysis) that isolates the causal direction between code quality and defect/time outcomes.
uninformedThe Future Work section raises whether a file's main-author-vs-minor-contributor status affects resolution-time predictability in low-quality code, but never connects this to co-author Tornhill's own earlier social code metrics (ownership, developer fragmentation/fractal value, knowledge maps) from 'Your Code as a Crime Scene' (reference 58) -- metrics that measure exactly this and are already present elsewhere in this corpus.
Read next: See extract-software-design-x-rays for the ownership/fragmentation/knowledge-map metrics that would directly operationalize this paper's own future-work hypothesis.
incompleteAll 39 codebases come exclusively from CodeScene's own customer base, and the paper cannot disclose selection criteria or codebase identities for confidentiality reasons. Companies that already pay for a code-quality tool are plausibly a non-random, quality-aware sample, which the paper notes as an external-validity threat but does not attempt to bound.
Read next: Look for an independent replication using a non-CodeScene-customer dataset or a different code-health/quality metric to check whether the 15x/124%/9x figures hold outside CodeScene's own customer base.