The Oracle (Referenced Evaluation)

An executable evaluation whose expected answer is referenced live from a running system and computed fresh per input (rather than authored and frozen), so it checks every input rather than a handful of scenarios and becomes the non-delegable fitness function of agent-driven building.

Also known as: the oracle, the then isn't authored, it's referenced, acceptance gate, byte-for-byte identical, fitness function the keep-the-candidate-that-survives game runs on, the objective and its evaluation

Part of: Evolutionary Architecture & Fitness Functions

How authors use it

Oldest first (extraction date when no publication date is known).

AuthorTheir termTheir meaning
What Survived Contact
Sam Ruby · 2026-07-01		the oracleThe evaluation of the objective — the fitness function the keep-the-survivor game runs on — that decides whether generated output is correct, and the one thing the person hiring the model cannot delegate back to it

Documents (2)

Propositions touching this concept (1)

By publication month, oldest first (extraction month when no publication date is known).

2026-07

Tensions

How far can day-2 operations be handed to self-healing / autonomous remediation?
from: What did the participants of the Thoughtworks 'Future of Software Engineering' Europe retreat (mid-2026) conclude about how agentic AI is reshaping software engineering?
Assessment: Ruby's report is descriptive (where practice actually stands) and Ocytko's is aspirational (where platforms could go); the gap between 'remediation is withheld' and 'failure modes disappear' is precisely the distance the optimists are projecting across. Bangser supplies the reason the gap may not close on efficiency grounds alone — the withheld human step is also the learning step.
Is the value question genuinely settled by evidence, or does uncertainty and sampling bias still dominate?
from: What did the participants of the Thoughtworks 'Future of Software Engineering' Europe retreat (mid-2026) conclude about how agentic AI is reshaping software engineering?
Assessment: The single sharpest divergence, and it sits directly under the headline whether-to-how convergence: they narrate the same shift but disagree on what it establishes. Edwards-Alexander and Herlein read the room's confidence as evidence; Vella and Ruby read the same room as a self-selected in-group. Ruby has the better of it methodologically — he is the only one to interrogate the sample and to notice the evidence is asymmetric (upside anecdotal, downside studied), which is uncomfortable precisely because the cohort's confidence is built on the weaker side.
Where is the locus of safety for non-engineers / citizen developers building software?
from: What did the participants of the Thoughtworks 'Future of Software Engineering' Europe retreat (mid-2026) conclude about how agentic AI is reshaping software engineering?
Assessment: Four different answers to the same governance question: the platform (Ocytko), the director's discipline (Ruby), the tooling interface (Bangser), or nobody-knows-yet (Ördög). Ruby and Ocytko are the true opposites — Ruby puts the burden on a human management skill, Ocytko designs the human out by making failure modes disappear in the platform. That both are held by attendees of one retreat is a measure of how unsettled the citizen-developer problem is.
Is code a durable artifact to be maintained, or an ephemeral byproduct regenerated from a more durable source (spec/objective)?
from: What is the future of software development in the agentic/AI age?
Assessment: The most-cited open question of the corpus, and genuinely unresolved (Herlein: 'no consensus here. None.'). The 'code is the better spec' counter is strong — code is unambiguous, executable, deterministic, whereas a prose spec is reinterpreted each run, which sits in tension with verifiability being the quality bar. Ruby's move is the most defensible: it dissolves the code-vs-spec fight by locating durability in the evaluation rather than any representation. Verraes and Ördög expose the load-bearing empirical unknown — nobody has data on regenerate-vs-patch cost or on long-term evolvability of agent-maintained code.
Where does the non-delegable human asset now live — the harness/platform, the oracle, the fundamentals, or the developer's understanding?
from: What is the future of software development in the agentic/AI age?
Assessment: This is the deepest structural disagreement — every author agrees a human residue remains but nominates a different residue, and the choice determines what to invest in and what to teach. The harness-centric (Edwards-Alexander), oracle-centric (Ruby), fundamentals-centric (Nilsson/Bangser), and understanding-centric (Fowler) camps are not merely emphases: they imply different career ladders, different hiring, and different failure modes. Ruby's oracle and Bangser's operational-learning are the most operationally concrete; Nilsson's is the least argued. The absence in the corpus of anyone who holds 'nothing is non-delegable' is itself telling (see unexamined assumptions).
How far should software be automated — toward the fully-autonomous dark/software factory, or held back to bounded autonomy with humans in the loop?
from: What is the future of software development in the agentic/AI age?
Assessment: The spectrum runs from Ördög/Ocytko (full automation is arriving and desirable) to Bangser/Ruby/Verraes (deliberately withhold the last step). The decelerationists have the better of the safety argument today — Ruby's 'nobody will cede the blast radius' and Bangser's 'removing humans removes learning' identify concrete, unmet prerequisites — but Ördög's context-dependence (backends, COBOL) shows the boundary is domain-specific, not absolute. The real disagreement is less about capability than about who absorbs the blast radius when an autonomous fix goes wrong.
Does the current evidence favor optimism or doubt about agentic engineering's payoff?
from: What is the future of software development in the agentic/AI age?
Assessment: The sharpest divergence in the corpus, because it is about the epistemic ground of every other claim. Nilsson's optimism is explicitly consensus-of-the-in-group and self-flagged as biased; Ruby and the Thoughtworks report both bring the only genuinely external evidence (DORA reversal, HBR intensification, outside surveys) — and it points the other way. On the current record, the skeptics have the stronger footing precisely because the optimistic case is methodologically weakest (a 'sample of one,' as Ruby says of his own best data). The whole cohort's cautious optimism may be an artifact of who was in the room.

Open gaps (2)