LLMs Corrupt Your Documents When You Delegate

Philippe Laban, Tobias Schnabel, Jennifer Neville · article · 2026-04-17 source ↗ Reader 🖥️

Delegated Editing & Document Corruption Evals Human-in-the-Loop LLM Limitations Vibe Coding

Current LLMs are unreliable delegates that silently corrupt documents during long delegated workflows, as demonstrated by the DELEGATE-52 benchmark across 52 professional domains and 19 models.

practical · argumentative — Empirical benchmark study with a clear argumentative thrust: current LLMs are unreliable delegates. Practical because it directly addresses the readiness of deployed AI systems for real-world delegation workflows.

Structure

Part	Role
Motivation: delegation as a new interaction paradigm	Sets the stakes — delegation (e.g., vibe coding) is emerging and requires trust
Introduction of DELEGATE-52 benchmark	Presents the evaluation instrument — 52 domains, long editing workflows
Main experimental finding: frontier models corrupt ~25% of content	Core evidence for the thesis
Aggravating factors: document size, interaction length, distractor files	Identifies conditions that worsen degradation
Negative result: agentic tool use does not help	Closes an obvious objection (maybe tools fix it)
Characterization: sparse but severe, silently compounding errors	Describes the failure mode qualitatively

Key terms

delegation Delegated Editing & Document Corruption
An interaction paradigm where users hand off document-editing tasks to an LLM with the expectation of faithful execution — distinct from conversational or co-editing use
DELEGATE-52 Evals
A benchmark simulating long delegated workflows requiring in-depth document editing across 52 professional domains
document corruption Delegated Editing & Document Corruption
Unintended degradation of document content introduced by the LLM during editing, measured as percentage of content corrupted
vibe coding Vibe Coding
Used as an example of delegated work — coding driven by high-level intent rather than precise specification

Propositions

explicitDelegation is emerging as a new interaction paradigm for LLMs in knowledge work.conf: high
explicitDelegation requires trust that the LLM will faithfully execute tasks without introducing errors.conf: high
explicitEven frontier models (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) corrupt an average of 25% of document content by the end of long workflows.conf: high
explicitNon-frontier models fail more severely than frontier models on DELEGATE-52.conf: high
explicitAgentic tool use does not improve LLM performance on delegated document editing tasks.conf: high
refinesMeta's new model is Muse Spark, and meta.ai chat has interesting new tools · p3 — Qualifies the harness-as-important-as-model claim: agentic tool use adds nothing on delegated document-editing fidelity.
explicitDocument degradation worsens with larger documents, longer interactions, and the presence of distractor files.conf: high
supportsHarness design for long-running application development · p1 — DELEGATE-52's finding that degradation worsens with longer interactions and larger inputs independently corroborates the long-running coherence-loss failure mode.
supportsAgents don't know what good looks like. And that's exactly the problem. · p4 — DELEGATE-52 finds degradation grows with document size, interaction length, and distractor files — empirical backing for the claim that more context degrades output.
explicitcorroboratedLLM-introduced errors are sparse but severe and silently compounding.conf: high
supports ←Bliki: Vibe Coding · p8 — Non-deterministic edits introducing regressions in unrelated code matches the sparse-but-severe, silently compounding error finding.
implicitcontestedCurrent LLMs are not ready to be trusted as autonomous delegates for document editing.conf: high
contradictsSkill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI · p4 — Benchmark evidence that autonomous delegation silently corrupts documents cuts against the advice to arrange work to be completely autonomous.
refinesPresentation: AI-First Software Delivery: Balancing Innovation with Proven Practices · p2 — The DELEGATE-52 finding that even frontier models silently corrupt long delegated workflows adds an empirical caution to the advice to progressively move toward unsupervised agents.
implicitdevelopedThe compounding nature of errors means that longer autonomous sessions are disproportionately dangerous, not just linearly worse.conf: medium
refines ←Agentics / Tech Things: Tokenmaxxing is dead, long live tokenmaxxing · p5 — theahura explicitly concedes the 'compounding error' regime was real ('up until recently... we called this compounding error') and claims a regime shift has since flipped it, bounding the 'longer autonomous sessions are disproportionately dangerous' claim in time rather than denying it. Note: his evidence for the flip is a single security-domain data point, so weight the refinement accordingly.
implicitcorroboratedHuman review checkpoints during long delegated workflows may be necessary to mitigate corruption.conf: medium
supports ←A cautionary tale about AI in tech writing: Why "Trust but Verify" is the only way to scale · p7 — Independently corroborates that autonomous LLM self-review/delegation is unsafe and that external checkpoints (here: deterministic tooling plus a separate human proofing pass) are needed to prevent silent corruption of long documents.

Arguments

Current LLMs are unreliable delegates for document editing workflows. strength: high

Explicit premises:

DELEGATE-52 simulates long delegated workflows across 52 professional domains
19 LLMs were tested on this benchmark
Even frontier models corrupt ~25% of document content by the end of long workflows
Non-frontier models fail more severely

Implicit premises:

25% content corruption is an unacceptable error rate for professional document editing
The 52 domains in DELEGATE-52 are representative of real-world delegation use cases
The simulated workflow lengths are representative of actual delegated sessions

Inferences:

If the best models corrupt a quarter of content, the technology is not ready for unsupervised delegation

The delegation problem is fundamental, not an artifact of tooling limitations. strength: medium