Coding Models Are Doing Too Much

nrehiew.github.io · article · 2026-04-20 source ↗ Reader 🖥️

Capability Provisioning (right-sizing to the task)Code Quality Coding Agents Evals Legacy & Organizational Forgetting Scope Discipline

Current coding models systematically rewrite code beyond the minimal fix required — a widespread, measurable failure mode called over-editing — and reinforcement learning can train models to produce faithful minimal edits without degrading their general coding ability.

mixed · argumentative — An empirical research report structured as an argument. The author identifies a practical problem (over-editing by coding models), constructs a benchmark to measure it, evaluates frontier models, and proposes a training method (RL) to fix it. Argumentative in that the entire piece advances a thesis about a specific failure mode and its remedy, but grounded in experimental evidence.

Structure

Part	Role
Problem definition: the Over-Editing problem	Names and illustrates the phenomenon — models modify code far beyond what is necessary to fix a bug, making diffs unrecognizable and code review harder
Green-field vs brown-field framing	Explains why over-editing matters: in brown-field work the existing code is deliberate, and the model's job is to fix only the issue
Benchmark construction and metrics	Creates a dataset of 400 programmatically corrupted problems with known minimal fixes, and introduces token-level Levenshtein Distance and Added Cognitive Complexity as metrics
Evaluation of frontier models	Shows that all frontier models over-edit, with GPT-5.4 worst and Claude Opus 4.6 best; demonstrates that high Pass@1 does not imply minimal editing
Prompting experiments	Shows that an explicit instruction to preserve original code reduces over-editing across all models, especially reasoning models
Reasoning vs non-reasoning comparison	Demonstrates that reasoning models over-edit more by default but respond better to explicit constraints
Training experiments (SFT, rSFT, DPO, RL)	Tests four training methods; SFT memorizes and collapses out-of-domain; RL generalizes, improves all metrics, and avoids catastrophic forgetting
LoRA and scaling experiments	Shows RL with LoRA nearly matches full fine-tuning, and the recipe scales from 4B to 14B parameters

Key terms

Over-Editing Scope Discipline
A model modifying code beyond what is strictly necessary to fix the problem at hand — the output is functionally correct but structurally diverges from the original more than the minimal fix requires
Token-level Levenshtein Distance
Edit distance computed over Python token sequences (not characters), normalized by total token count, measuring how much a model's patch diverges from the minimal fix
Added Cognitive Complexity
The absolute difference in Cognitive Complexity between the model's output and the original uncorrupted code; any nonzero value means the model introduced unnecessary structural complexity
brown-field Legacy & Organizational Forgetting
Working within an existing codebase where code has been deliberately written and understood by the team, as opposed to green-field (building from scratch)
relative patch score
The ratio comparing the model's edit distance (from corrupted input) to the ground-truth minimal edit distance, where values closer to zero indicate the model's patch resembles the true minimal fix

Propositions

explicitCurrent coding models systematically modify code beyond what is necessary to fix bugs, a behavior the author calls Over-Editing
supportsArchitecture as Code to Teach Humans and Agents About Architecture · p5 — Measured over-editing gives empirical backing to the claim that LLMs default to doing more than necessary unless explicitly constrained.
explicitOver-editing is invisible to test suites because the rewritten code is still functionally correct
explicitOver-editing makes code review dramatically harder and risks quiet degradation of codebase quality
explicitdevelopedPass@1 is necessary but not sufficient for evaluating coding models — it fails to capture how much a model changed beyond what was required
extends ←Local AI · p9 — Loukides generalizes the Pass@1-insufficiency finding: published benchmarks broadly are unreliable for model selection, requiring domain-specific evaluation.
explicitGPT-5.4 over-edits the most among frontier models while also being one of the weakest on correctness
explicitClaude Opus 4.6 achieves the highest Pass@1 of any model evaluated while also producing the smallest diffs
explicitA simple explicit prompt instruction to preserve the original code reduces over-editing across all models
explicitThe constraint to make minimal edits inadvertently narrows the search space, steering models toward more correct fixes
explicitReasoning models over-edit more by default than their non-reasoning counterparts but respond better to explicit constraints
explicitOver-editing in reasoning models is a default behavior that can be overridden, not a fundamental limitation
explicitSFT on programmatically generated data memorizes specific corruption reversals rather than learning general minimal editing behavior
explicitReinforcement learning is the only training method that generalizes minimal editing behavior to out-of-domain corruptions without catastrophic forgetting
explicitcorroboratedThe degree of catastrophic forgetting is proportional to the difference between the model's original distribution and the training data distribution
supports ←The Post-training Process OpenAI Used for ChatGPT · p6 — OpenAI's practice of mixing pretraining data into the SFT and RL objectives to prevent catastrophic forgetting is a concrete real-world instance consistent with this proposition's general distributional-distance account of forgetting.
explicitFor style-level behavioral changes where the underlying capability already exists, LoRA is likely sufficient and considerably cheaper than full fine-tuning
implicitWriting more tests is not a sufficient mitigation for over-editing because over-editing is invisible to test suites
implicitHigh correctness and minimal editing are not in tension — the best-performing models on Pass@1 also tend to produce smaller diffs

Arguments

All frontier coding models over-edit, with significant variation across model families

Explicit premises:

A benchmark of 400 programmatically corrupted problems with known minimal fixes was constructed
Token-level Levenshtein Distance and Added Cognitive Complexity were measured for all frontier models
GPT-5.4 scored 0.395 Levenshtein (reasoning) while Claude Opus 4.6 scored 0.060

Implicit premises:

Programmatic corruption with known reversals is a valid proxy for real-world minimal bug fixes
The selected models are representative of frontier coding capability

Inferences:

S
i
n
c
e
a
l
l
m
o
d
e
l
s
s
c
o
r
e
a
b
o
v
e
z
e
r
o
o
n
b
o
t
h
m
e
t
r
i
c
s
,
a
l
l
o
v
e
r
-
e
d
i
t
t
o
s
o
m
e
d
e
g
r
e
e
;
s
i
n
c
e
s
c
o
r
e
s
v
a
r
y
w
i
d
e
l
y
,
m
o
d
e
l
a
r
c
h
i
t
e
c
t
u
r
e
a
n
d
t
r
a
i
n
i
n
g
c
h
o
i
c
e
s
m
e
a
n
i
n
g
f
u
l
l
y
a
f
f
e
c
t
e
d
i
t
i
n
g
s
t
y
l
e

Pass@1 alone is an inadequate evaluation metric for coding models

Explicit premises:

A model can score perfectly on Pass@1 while completely rewriting every function it touches
Over-editing is functionally correct and invisible to test suites

Implicit premises:

Code quality encompasses more than functional correctness — reviewability, maintainability, and minimal surprise matter

Inferences:

Over-editing is a default behavior, not a capability limitation

Explicit premises:

Adding an explicit instruction to preserve original code reduces Levenshtein Distance for every model tested
Every model except DeepSeek R1/v3 also improves Pass@1 with the explicit prompt

Implicit premises:

If a model can produce minimal edits when prompted, the capability exists — the issue is in the default behavior

Inferences:

T
h
e
m
o
d
e
l
s
a
l
r
e
a
d
y
p
o
s
s
e
s
s
t
h
e
a
b
i
l
i
t
y
t
o
m
a
k
e
m
i
n
i
m
a
l
e
d
i
t
s
;
t
h
e
p
r
o
b
l
e
m
i
s
w
h
a
t
t
h
e
y
d
o
w
h
e
n
u
n
c
o
n
s
t
r
a
i
n
e
d

Extended reasoning both causes and can cure over-editing

Explicit premises:

Reasoning models over-edit more than non-reasoning counterparts in the generic (unconstrained) setting
In the explicit setting, reasoning models match or undercut non-reasoning counterparts on Levenshtein Distance
The gap between generic and explicit settings is consistently larger for reasoning models

Implicit premises:

Extended reasoning without constraint gives models room to 'improve' code that doesn't need improving
The same reasoning capacity enables better instruction-following when constraints are given

Inferences:

RL is the best training method for teaching minimal editing behavior

Explicit premises:

SFT collapses to 0.458 Pass@1 on out-of-domain corruptions (memorization)
rSFT and DPO show only slight improvement over the 8-shot baseline
RL improves all three metrics over both baselines on out-of-domain data
RL shows no degradation on LiveCodeBench (general coding ability preserved)

Implicit premises:

A training method that generalizes out-of-domain without catastrophic forgetting is preferable to one that performs well only in-domain

Inferences:

Catastrophic forgetting is proportional to distributional shift between training data and the model's original distribution

Explicit premises:

SFT (trained on programmatic data far from model's distribution) shows 43% LiveCodeBench degradation
rSFT and DPO (trained on self-distilled data closer to model's distribution) show slight degradation
RL shows no degradation

Implicit premises:

RL's inherent bias toward KL-minimal solutions keeps the trained model close to the original distribution
Self-distilled data is closer to the model's distribution than programmatically generated data

Inferences:

LoRA is sufficient for teaching minimal editing behavior

Explicit premises:

LoRA at rank 64 nearly matches full RL on Levenshtein Distance and beats it on Added Cognitive Complexity
LiveCodeBench shows essentially no degradation at rank 64
The base model already has the capability to edit code and fix bugs

Implicit premises:

Style-level behavioral changes require fewer parameter updates than capability acquisition

Inferences:

Narratives

Questions addressed

⭕ Do current coding models over-edit, and can this be measured?
⭕ Can prompting alone fix over-editing?
⭕ Does reasoning make over-editing worse?
⭕ Can models be trained to produce minimal edits?
⭕ Is full fine-tuning necessary for this behavioral change?

Gaps

incompleteThe benchmark uses only single-function, single-bug corruptions. Real-world brown-field edits often span multiple files, involve multi-step reasoning, and require understanding broader architectural context. The author acknowledges this but does not address how the metrics or training approach would scale to more realistic settings like SWE-Bench Pro.
incompleteThe corruptions are all value-level (operator flips, boolean swaps) rather than structural (control flow, API misuse, concurrency bugs). It is unclear whether the over-editing phenomenon and the RL remedy hold for structural bugs where the minimal fix may itself be structurally complex.
incompleteThe paper does not investigate whether over-editing correlates with RLHF training that optimizes for helpfulness. Models may be trained to be maximally helpful, which in a coding context manifests as 'improving' code proactively. This would be a root-cause analysis the paper does not attempt.
incompleteNo analysis of how the system prompt, context length, or conversation history affects over-editing. In practice, coding agents operate with long contexts and multi-turn interactions — does over-editing get worse with more context?
incompleteThe RL reward function combines correctness and Levenshtein distance with a fixed weighting (pass bonus +0.1, fail penalty -0.2). No sensitivity analysis on these hyperparameters is provided, and the Pareto frontier between correctness and minimality is not explored.

Closes gaps elsewhere

LLMs Corrupt Your Documents When You Delegate · g2 — Supplies the missing mitigation strategies for unrequested-modification corruption: explicit preserve-the-original prompt instructions and RL training that generalizes minimal-editing behavior.

Coding Models Are Doing Too Much

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Closes gaps elsewhere

Related documents