A cautionary tale about AI in tech writing: Why "Trust but Verify" is the only way to scale

Betsalel (Saul) Williamson · article · 2026-07-13 Reader 🖥️

practical · first-person practitioner case study wrapped around a prescriptive step-by-step framework, published as a LinkedIn article with an appended reusable prompt template — The piece is not an argument about AI-and-trust in the abstract; it is a delivery-methodology writeup (specs, tooling, review stages) offered as a transferable recipe, with the personal anecdote (17,500 words / 8 hours) functioning as the credibility hook rather than as evidence in its own right.

Structure

Key terms

• Trust, but Verify
  specifically the 5th and final pipeline stage: after specs, shards, and deterministic tooling have handled the mechanical work, the human spends the freed-up time on reading/comprehension/procedure-testing of the residual content
• slop AI Slop
  the predictable output of blindly trusting a probabilistic, non-deterministic process without quality controls -- bad design, code, or docs
• shard / sharding
  splitting a large Markdown manual into small, maintainable pieces (via the BMAD method and @kayvan/markdown-tree-parser) so the LLM works in small batches and a scripted compiler reassembles them with corrected heading hierarchy
• ground truth
  the specification documents (optionally extracted directly from the codebase) that constrain the AI so it never has to guess
• the tools pipeline
  the named, deterministic layer (Prettier, markdownlint-cli2, custom static analysis, Vale with a custom dictionary) that catches formatting drift, broken links, and voice/terminology violations without relying on the LLM to check itself
• Specs -> Technical Docs -> Client Docs
  a required three-stage translation cascade: rough-but-correct specs, then validated technical docs, then jargon-free client docs derived only from the validated technical docs

Propositions

• explicitAI can accelerate drafting and formatting, but it cannot validate a procedure or business logic for intent, make stylistic UX/design choices addressing real customer problems, know what not to include to prevent confusion, or be accountable for the final product.conf: high
• explicitcorroboratedBlindly trusting the direct output of an LLM will ship bad design, code, or docs, because it is a probabilistic and non-deterministic process without quality controls.conf: high
  extendsLeading Open Source Author Calls for Verification over Trust in Software Supply Chains · p1 — Applies the general 'verification over trust' security principle (originally about software supply chains) to a different domain: technical documentation production, where the risk is not a malicious actor but the LLM's structural inability to validate or own its own output.
  supports ←You Probably Won’t Read This Article…and That’s OK · p7 — Williamson's warning that raw LLM output must never be trusted directly, but wrapped in a deterministic verification pipeline, illustrates exactly why 'ask another LLM to gate the content' can't work as stated — the validating work has to be structurally separated from the generating model, echoing this document's turtles-all-the-way-down concern.
• explicitTreating an AI-assisted documentation project exactly like a high-quality software development process is what let the author get AI's speed without risking a reputation for delivering slop.conf: high
• explicitAI should never be allowed to guess; work must always start from a structured specification document, worked one feature at a time in small batches.conf: high
  supportsSam Hatoum's Post · p9 — Both treat specs as durable external memory/ground truth that constrains an AI implementer/drafter lacking persistent memory of design intent.
• explicitClient-facing documentation should only be written after the technical docs are drafted and validated, because the validated technical docs become the reliable basis for a jargon-free translation.conf: high
• explicitAmbiguous terms and edge cases must be explicitly spelled out in the specs -- both because natural language is ambiguous between humans, and because the AI needs an unambiguous ground truth to use the right domain terminology consistently.conf: high
• explicitAn LLM should never be relied on to proofread its own work; that job belongs to a strict, deterministic tools pipeline (formatters, linters, a style checker with a custom dictionary).conf: high
  supportsLLMs Corrupt Your Documents When You Delegate · p10 — Independently corroborates that autonomous LLM self-review/delegation is unsafe and that external checkpoints (here: deterministic tooling plus a separate human proofing pass) are needed to prevent silent corruption of long documents.
• explicitSplitting a large manual into small 'shards' improves LLM output quality, mirroring the software-delivery discipline of small batch sizes.conf: medium
• explicitestablishedOnce specs are drafted, drafting is expanded by AI, and formatting is handled by tooling, the bottleneck shifts to human reading, comprehension, and procedure-testing speed -- i.e., verification, not generation, becomes the scarce resource.conf: high
  extendsSam Hatoum's Post · p6 — Both claim the binding constraint has shifted from generation/implementation speed to verification capacity -- Hatoum's post makes the claim for AI-assisted coding; this document applies the identical structural claim to AI-assisted technical documentation.
  supports ←When Content Is Free, Trust Is the Product · p9 — Both identify that once production is cheap (via AI), verification/trust — not generation — becomes the scarce, bottlenecked resource.
  supports ←Article: When Spec-Driven Development Pays Off · p1 — An independent domain (technical writing, not code) converges on the same claim that verification, not generation, is now the scarce resource.
• implicitThe headline 8-hour/17,500-word production claim is not evidence that AI eliminates the traditional workload of rigorous technical writing, but evidence that a disciplined pipeline reallocates that workload (into spec-writing, tooling setup, and targeted human review) rather than removing it.conf: medium
• implicitcorroboratedFinal accountability for documentation quality remains irreducibly human no matter how much of the mechanical work is automated.conf: high
  supports ←Should You Still Become a Software Engineer in 2026? GitHub VP · p3 — Both locate final accountability irreducibly with the human regardless of how mechanically the work was produced, independently converging on the same 'trust but verify' principle.
• implicitBy publicly correcting a terminology error ("Spec Flow" vs. the unrelated Specflow framework) after a colleague flagged it, the author's own post-publication edit performs the 'trust but verify' principle the article argues for.conf: medium
• implicitThe described 5-step pipeline is intended as a generalizable, repeatable method across projects and teams, not a one-off artifact of this particular engagement -- offered as a literal copy-paste prompt template.conf: high

Arguments

Narratives

• The author, a veteran technical writer, recently delivered a ~17,500-word technical manual to a client in eight hours using an LLM. Because he has spent years doing the traditional, painstaking version of this work, he treats his own fast AI-assisted output with suspicion rather than pride, and narrates the specific five-step process (specs, ambiguity resolution, sharding plus automated tooling, human proofing, and final 'trust but verify' review) he used to keep the output trustworthy. He closes by offering the process as a literal, reusable prompt for other teams, then appends a post-publication correction after a colleague pointed out a naming collision with an existing framework (Specflow).
  → Serves as both proof-of-concept and structuring device for the whole piece: the personal narrative supplies the ethos (rigor-minded veteran, not an AI hype-poster), the step-by-step retelling supplies the transferable mechanism, and the closing self-correction quietly re-enacts the article's own thesis -- that even careful work benefits from an external check.

Questions addressed

• ✅ Can AI safely be used to produce large volumes of technical documentation quickly?
  Yes, but only inside a disciplined pipeline -- never by trusting the LLM's raw output directly.
• ✅ What specifically can't AI do in a documentation workflow, that a human must still do?
  Validate procedures/business logic, make UX/design judgment calls, decide what to omit, and bear accountability for the final product.
• ✅ How do you keep an AI-drafted corpus consistent and low-ambiguity at scale?
  Explicit specs with defined terms and edge cases, small-batch sharding, and a deterministic tools pipeline (formatting, linting, style enforcement) rather than relying on the LLM to self-correct.
• ⭕ Is '17,500 words in 8 hours' actually a fair or meaningful productivity comparison against traditional (non-AI) technical writing?
  The piece never benchmarks against a manual-writing baseline, nor breaks down how the 8 hours were allocated across the five steps.
• ⭕ Does the 'Trust but Verify' step itself scale, or does verification time grow with document size/complexity the same way manual writing effort would -- potentially eroding the claimed speed advantage as documents get larger?
  Not addressed; the piece asserts that verification becomes 'the bottleneck' but gives no data on how that bottleneck scales with document size.

Gaps

• incompleteThe 'Trust but Verify' framing itself is a well-worn pattern already pervasive in AI-adoption writing (borrowed from the Reagan-era arms-control phrase and repeated near-verbatim across countless similar LinkedIn/AI-adoption pieces). What is genuinely more specific here is not the slogan but the mechanism underneath it: a concrete 5-stage pipeline with named, checkable tools (Prettier, markdownlint-cli2, custom static analysis, Vale with a custom dictionary, the BMAD method, @kayvan/markdown-tree-parser) and an explicit spec-to-client-doc translation cascade. A reader already saturated with 'trust but verify' framings should judge this piece on whether that tooling/mechanism is reusable, not on the slogan, which adds no new information by itself.
  Read next: Compare against other AI-adoption pieces the reader has already logged with the same 'trust but verify' framing to see whether any of them supply comparably concrete, named tooling rather than just the aphorism.
• incompleteThere is no quantification of how the claimed 8 hours were actually allocated across the five steps -- in particular, no data on how long step 5 ('Trust but Verify', the step doing all the safety work) took relative to spec-writing, drafting, and tooling setup. Without that breakdown, a reader cannot tell whether verification was proportionate to the claimed output volume or was itself compressed to fit the '8 hours' headline -- exactly the kind of unfalsifiable single-anecdote claim that should trigger skepticism by the piece's own stated logic ('If you blindly trust... you are going to ship bad design, code, docs').
  Read next: Look for practitioner accounts of AI-assisted technical writing that report time-per-stage or word-count-per-hour broken out by drafting vs. verification, rather than a single aggregate figure.
• illogicalThe piece's own thesis is 'never let AI guess' and 'never rely on an LLM to proofread its own work' -- yet the described QC layer is dominated by deterministic formatting/style tools and a single human 'proofing for flow' pass, with no description of independent fact-checking of the underlying technical claims/procedures against the actual system. The step that supposedly matters most (procedure validation, checking for 'odd assumptions in the code,' 'terms overlapping or in conflict with end-user language') is described impressionistically as a bullet list of questions, while the mechanical tooling steps are described with brand-name, checkable specificity. This asymmetry suggests the namesake verification step is the least rigorously engineered part of the pipeline it is meant to anchor.
  Read next: Look for accounts that specify concrete verification methods for procedure/business-logic correctness (e.g., running the documented steps against the live system, second-reviewer sign-off) rather than describing verification only as an unstructured reading pass.
• uninformedThe framework assumes the specs themselves are correct ground truth ('as long as they are correct'), but does not address how errors introduced at the spec-writing stage -- itself sometimes AI-assisted ('extract this ground truth directly from the codebase') -- get caught, given that the human verification stage (step 5) happens downstream of specs, drafts, and tooling, and is described as targeting prose/flow/terminology issues rather than spec-accuracy issues.
  Read next: Look for material specifically on validating AI-extracted specifications against source systems, not just validating prose derived from specifications.
• incompleteThe piece treats the shift of the bottleneck from generation to verification as a personal discovery specific to documentation, without situating it in the broader pattern -- already present elsewhere in this corpus -- that verification capacity, not implementation speed, is the new binding constraint across AI-assisted knowledge work generally (code included). The documentation-specific framing adds real value (named tools, a shard-based pipeline) but doesn't acknowledge it is one instance of a wider, already-observed shift.
  Read next: Cross-read against corpus material on spec-driven software development making the identical claim for code, to see whether the same verification-capacity bottleneck has a consistent shape (or different failure modes) across code vs. documentation.

Related documents

• Leading Open Source Author Calls for Verification over Trust in Software Supply Chains
• LLMs Corrupt Your Documents When You Delegate
• 11 Tips For AI Coding With Ralph Wiggum
  Feedback Loops Guardrails Human-in-the-Loop
• Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation
  Feedback Loops Guardrails Human-in-the-Loop
• [FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero
  Feedback Loops Human-in-the-Loop Spec-Driven Development
• This coding agent doesn’t want your feedback — it ships without it
  Guardrails Human-in-the-Loop Verifiability (as the Quality Bar)
• No Vibes Allowed: Solving Hard Problems in Complex Codebases – Dex Horthy, HumanLayer
  AI Slop Human-in-the-Loop Spec-Driven Development
• How we Claude Code
  Human-in-the-Loop Spec-Driven Development Verifiability (as the Quality Bar)
• What Survived Contact
  Guardrails Human-in-the-Loop Spec-Driven Development
• Addy Osmani's Post
  Feedback Loops Guardrails