What Survived Contact

Sam Ruby · article · 2026-07-01 source ↗ Reader 🖥️

practical · mixed — A retired practitioner's field report from a 'future of software development' retreat, blending narrative reportage of the room with an argumentative reframe of how non-engineers should direct AI coding agents and expository detail on evaluating objectives (BDD, the referenced oracle).

Structure

Key terms

• survived contact
  Which of the author's ideas held up, got argued with, and stayed standing when stress-tested against a room of experts who knew the subject better than he did
• the spine
  The core finding the author cares most about and keeps arguing: rigor doesn't vanish when the agent writes the code, it migrates upstream into specs and down into tests, types, constraints, and risk-tiering
• a hire, not a tool
  The claim that when a manager reaches for an LLM instead of routing work to their team, they have added a participant to be managed, not picked up an instrument to operate
• manage by objective (not by method) Manage by Objective (not by Method)
  Drucker's 1959 principle applied to agents: when the worker is out-known about the specifics, you supply a goal it can test itself against rather than prescribing how to do the work
• the oracle The Oracle (Referenced Evaluation)
  The evaluation of the objective — the fitness function the keep-the-survivor game runs on — that decides whether generated output is correct, and the one thing the person hiring the model cannot delegate back to it
• BDD / specification by example
  Stating the objective AS its own executable evaluation: a Given/When/Then scenario that is the behavior written so it runs, rather than a description sitting next to a separate test
• referenced (not authored) then
  The 'then' of an acceptance scenario computed live from a running reference (e.g. requiring byte-for-byte identity with the existing Rails app) per input, instead of a hand-written, frozen expected result
• sample of one Product Discovery & Experimentation
  The author's single-project, single-agent, single-retired-developer anecdote whose edges he keeps marking; also turned on the retreat itself, which is a sample too and not a random one
• software darwinism
  Standing up several candidate implementations at once and keeping the one that survives the oracle, rather than iterating serially, once building is cheaper than deciding what to build

Propositions

• explicitestablishedWhen the agent writes the code, rigor does not vanish; it migrates upstream into specifications and down into test suites, type systems, constraints, and the tiering of code by how much damage a mistake could do.conf: high
  supports ←Podcast: Formal Methods for Every Engineer in an AI-Powered Future · p18 — Concrete instance of 'rigor migrates upstream into specifications and down into test suites/constraints' — the formal spec is the upstream rigor, model-based testing the downstream test-suite instrumentation.
  supports ←Software Design in the Agentic Age: Placing Your Bets · p8 — Independent citation of the same Sam Ruby 'rigor migrates' framing already central to what-survived-contact's own thesis; both retreat write-ups treat this as the settled cross-retreat finding.
• explicitTwo retreats twenty-one weeks and a continent apart independently landed on the same 'rigor migrates' finding, which is about as close to a settled result as a field this young produces.conf: high
• explicitThe scope of the conversation expanded beyond the engineering organization into economics, ethics, and geopolitics (sovereignty, regulation, antitrust, environmental and human costs, unpaid maintainers), partly because the European venue drew a more international crowd.conf: medium
• explicitForecasts placed a year or three out — a supervisory layer, agents as first-class org-chart participants, old semantic machinery grounding domain-aware agents — arrived here carrying production numbers instead of speculation, compressing fastest where the earlier room had been most cautious.conf: medium
• explicitcorroboratedSelf-healing systems have moved only halfway: diagnosis is automated while remediation is withheld from the machine, because letting a machine act on a fix unsupervised hands it the blast radius, and no one is yet ready to cede that.conf: high
  refines ←Automated Bug Fixing at Facebook Scale · p9 — extract-what-survived-contact#p5 frames self-healing systems as having 'moved only halfway' because remediation itself is withheld from the machine. SapFix complicates that framing: as early as 2018, remediation (candidate patch generation and autonomous testing) was already fully automated at industrial scale — what was withheld was specifically the deployment/merge decision, a narrower and more precise line than 'remediation' generally.
  supports ←Cloudflare Turns CI Pipelines into TypeScript Workflows · p6 — Cloudflare's self-healing example — an agent proposes and commits a patch but leaves the run failed until a human merges — is a concrete instance of automating diagnosis while withholding remediation authority from the machine.
• explicitcorroboratedOptimism and worry are now both backed by data, but asymmetrically: the optimism rests on uncontrolled, single-team, self-reported metrics while the worry leans on outside studies and surveys, so the doubt is currently better evidenced than the hope.conf: high
  supports ←GitLab 19.2 Puts AI Agents to Work on the Security Backlog · p8 — The unqualified 400% ROI figure is another instance of the corpus's asymmetry: optimism resting on uncontrolled, vendor-commissioned metrics rather than outside studies.
• explicitA gathering of invited, senior practitioners convened by a consultancy is a non-random sample that tilts the agenda toward governing, verifying, and risk-tiering rather than disruption, because it over-represents large incumbents and their advisors.conf: high
• explicitWhen a manager reaches for an LLM instead of routing work to the team that reports to them, they have made a hire, not picked up a tool — so the 'may non-engineers steer the model?' permission question dissolves into 'do they know how to manage by objective?'conf: high
• explicitcorroboratedDirecting an agent well means managing by objective rather than by method (Drucker, 1959): because the director is out-known about the specifics, they should supply a goal the agent can test itself against instead of prescribing the method.conf: high
  supports ←Slack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation · p3 — Expressing a test as an objective the agent tests itself against, rather than a prescribed action sequence, is a concrete instance of directing an agent by objective, not by method (Drucker).
• explicitA model handed a task feels stakeless, but a model handed a goal it can test itself against behaves like a stakeholder; the stake was never missing from the worker — it lives in the objective, in a form you can actually check.conf: high
• explicitManage-by-objective only works when the objective is stated so it can be checked; an objective you can't evaluate is a wish, i.e. the same stakeless task to be avoided.conf: high
• explicitBDD / specification by example is uniquely valuable because a Given/When/Then scenario is the objective stated as its own runnable evaluation, whereas prose specs can't run, types are inferable, and constraints and risk tiers are downstream.conf: high
• explicitcorroboratedAn acceptance oracle whose expected answer is referenced live from a running system (computed fresh per input) rather than authored and frozen covers inputs never enumerated — the difference between checking forty scenarios and checking every request there is.conf: high
  extends ←Software Factories, Light and Dark · p15 — Generalizes the single 'live-referenced acceptance oracle' case into a broader taxonomy (green/red tests, type gates, property tests, rubric-based review agents) of what counts as a trustworthy, hard-to-fake automated check.
  supports ←How Datadog Used Claude and Cursor for Test-Driven Production Migration · p4 — Datadog's blue/green validator is a textbook real-world instance of the referenced-live acceptance oracle: expected answers are referenced fresh from the running old system per request rather than authored and frozen, so it covers every live input rather than a handful of enumerated scenarios.
• explicitcorroboratedThe objective and its evaluation — the oracle — is the one thing the person hiring the model cannot hire out; owning it is what makes the act a hire and not a prayer.conf: high
  extendsAgent Harness Engineering · p7 — Agent-harness-engineering argues evaluation should be separated from generation into a distinct agent; this doc pushes the separation up to the human hirer, arguing the objective's evaluation (the oracle) is the one thing the person directing the model cannot delegate at all.
  supports ←Slack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation · p6 — Slack keeps the pass/fail assertions engineer-defined while delegating only the path — the oracle stays human-owned, exactly the 'objective and its evaluation cannot be hired out' claim.

Arguments

Narratives

• The author kept his laptop open with Claude coding in the background throughout the retreat's sessions, and when an incredulous participant probed how he was using it he showed live examples straight from the work in progress.
  → Grounds the 'sample of one' in demonstrable practice and lets the ideas be stress-tested with evidence rather than assertion.
• In the 'bring me a rock' session, the room narrowed exploration-by-elimination from how to explore to who should even be allowed to, surfacing the worry that non-engineers steering the model produce slop.
  → Sets up the reframe that directing a model is a hire, not a permission-gated tool-use.
• Roundhouse — the author's third rebuild of the same problem — used an acceptance gate that fetched the same URL from Rails and from each generated target and required byte-for-byte identical responses, queried live per request and never frozen.
  → Concrete demonstration of the referenced oracle: the objective evaluated over every input, not a handful of authored scenarios.
• 'We're regulated; we can't afford the risk' was the exact sentence the author heard about open source nearly thirty years ago, from someone whose call was defensible but who was guarding the wrong artifact — governance never lived in inspecting every line but in governing what the code answered to.
  → Analogical argument that governance relocates rather than disappears, predicting the same relocation for agent-written code.
• Three people on three continents — matz, Ori Pekelman, and the author — have each been carrying a substantial piece of a system, coordinating through nothing more exotic than issues and pull requests.
  → Offers evidence for the open question of whether large systems can be composed from small teams that never merge into one.

Questions addressed

• ✅ How much of the author's 'sample of one' thesis survived contact with a room that knew the subject better than he did?
  The spine (rigor migrates, doesn't vanish) held and matched the earlier retreat; his ideas were sought out, argued with, and taken seriously, and the near-term engineering forecast had aged fast toward his optimism.
• ✅ Given that expertise separates a good outcome from slop, should non-engineers be allowed to steer the model at all?
  Wrong question: directing a model is a hire, not a tool-use, so you don't forbid it — you require the director to own the objective and its evaluation (manage by objective).
• ✅ Is a peer without stake (the model) a diminished peer?
  No — the stake was never missing from the worker; it lives in the objective. Manage by task and its absence is real, but manage by objective and the objective supplies the stake in a checkable form.
• ✅ How should an objective be evaluated so that manage-by-objective actually works?
  State the objective as its own runnable evaluation (BDD), and where a running reference exists, use a referenced oracle that computes the expected answer fresh per input rather than authoring and freezing answers.
• ⭕ Once building a thing costs less than deciding whether to build it, why iterate serially — and when do you evolve an implementation versus start over?
  Left open: he floats 'software darwinism' (stand up several candidates, keep the survivor) and notes restarts are cheap when the oracle stays constant and prior code becomes a corpus to consult, but doesn't know where the evolve-vs-restart line falls.
• ⭕ How short does the list of things only a human can do get as best practices (code review, security audit, testing, docs, triage, postmortems) move to the machine?
  Left open: the register flipped fast and even code review and security audit went early; his bet is that when the list stops shrinking, the oracle is the only thing left on it.
• ⭕ Can large systems be composed from small ones held by small teams that never merge, coordinating only via issues and pull requests?
  Left open as a line to add to the map, backed only by a three-person, three-continent anecdote.

Gaps

• incompleteThe whole cross-retreat comparison rests on an acknowledged seam: the author read only the public summary of the earlier retreat, so he cannot separate 'the conversation genuinely grew' from 'the earlier summary simply compressed this out.' The central 'what changed' claim is offered on face value with that limitation showing.
  Read next: Obtain the fuller record (not just the public key-takeaways summary) of the earlier Thoughtworks retreat and re-run the comparison against like-for-like source material.
• incompleteThe optimistic evidence is uncontrolled, single-team, self-reported metrics (a sample of one), which the author admits carries no conclusions; the piece names the asymmetry but does not offer a path to controlled, comparable evidence for the upside.
  Read next: Look for controlled or quasi-experimental studies of agentic-coding productivity and quality, and for methods that make single-team metrics comparable across teams.
• incompleteThe referenced-oracle technique is demonstrated only where a running reference implementation already existed (rebuilding Rails behavior in Roundhouse). The piece does not address how to construct a self-generating oracle for genuinely new behavior, where the intent lives only in a human's head and there is nothing to reference.
  Read next: Read on property-based and metamorphic testing, and on deriving executable oracles for green-field behavior where no reference system exists.
  closed byPodcast: Formal Methods for Every Engineer in an AI-Powered Future — g3 asks how to build a self-generating oracle for genuinely NEW behaviour, where no running reference implementation exists to reference an expected answer from live. Formal methods supply exactly that: instead of computing the expected answer from a pre-existing running system, the engineer authors an abstract formal model of the intended behaviour and uses model-based testing to replay the model's behaviours against the implementation — the authored model itself becomes the oracle, constructed from understanding before any reference implementation exists. (Note the mechanism difference from the referenced-oracle case: the answer is authored, not referenced live.)
• illogicalThe claim that the model's missing stake 'lives in the objective' assumes the checkable oracle fully captures what the human cared about. It doesn't examine the Goodhart case where a passable oracle diverges from the true, uncheckable intent — the model satisfies the letter of the objective while missing its point.
  Read next: Read on specification gaming, Goodhart's law, and reward hacking in agentic systems, and on how to detect an oracle that has drifted from intent.
• incompleteThe 'small teams that never merge into one' scaling hypothesis rests on a three-person anecdote and does not address where plain issues-and-PRs coordination breaks down as the number of subsystems, their coupling, or team count grows.
  Read next: Read on coordination cost, Conway's law limits, and the failure modes of loosely federated subsystems at larger scale.

Related documents

• Agent Harness Engineering
• AI Is Changing the Rules of Entrepreneurship
  Human as Orchestrator of Agents Near-Zero-Cost Code Product Discovery & Experimentation Small Teams & Compression
• Article: Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules
  Evolutionary Architecture & Fitness Functions Guardrails Human-in-the-Loop
• Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident
  Data Sovereignty Guardrails Human-in-the-Loop
• Addy Osmani’s Post
  AI Employee vs Tool Framing Human as Orchestrator of Agents Human-in-the-Loop
• The Founder's Playbook: Building An AI-Native Startup
  Human as Orchestrator of Agents Near-Zero-Cost Code Small Teams & Compression
• Gene Kim on Vibe Coding - Part III | Episode 412 | Troubleshooting Agile
  Guardrails Small Teams & Compression Spec-Driven Development
• The Future of Work by Henrik Kniberg
  AI Employee vs Tool Framing Human as Orchestrator of Agents Human-in-the-Loop
• A cautionary tale about AI in tech writing: Why "Trust but Verify" is the only way to scale
  Guardrails Human-in-the-Loop Spec-Driven Development
• 11 Tips For AI Coding With Ralph Wiggum
  Guardrails Human-in-the-Loop