Presentation: The Multi-Agent Approach: Building Reliable and Controllable Software Development Automation

Itamar Friedman · rss · 2026-07-08 source ↗ Reader 🖥️

practical · Practitioner conference talk (QCon AI / InfoQ presentation) with audience Q&A — a how-to argument for architecting AI software automation, grounded in vendor and third-party survey data plus product demos. — Spoken transcript of a live talk by Qodo's founder/CEO; propositions are stated informally and restated across the arc, then normalized here. The speaker is also selling a product (Qodo), so many statistics are vendor-sourced — treated as evidence with an interest caveat, not as neutral measurement.

Structure

Key terms

• Arbiter / arbitration
  A designated authority (agent or human) that decides which output is correct when agents disagree — rival implementations of the same prompt, a planning agent vs a coding agent, or a testing agent vs a coding agent. Friedman insists it can only adjudicate if the plan/spec carries enough context, so the arbiter first checks that the ticket is complete enough to arbitrate.
• Context engine
  No longer a RAG lookup but an agent in its own right (ingestion, queue and retrieval sub-agents) specialized in fetching the right multi-source context — code versions, PR history, in-PR review knowledge, org rules, docs and logs — with a meaningful share (~8%) coming from non-code sources like ARCHITECTURE.md.
• Productivity ceiling / glass ceiling
  Successive limits on how much value AI dev tooling delivers (plotted as gain on the y-axis vs investment on the x-axis); each is broken only by investing in the next capability layer, and even code-review/quality workflows hit a further ceiling that demands an adaptive, quality-grounded multi-agent system.
• Multi-agent vs workflow
  Friedman deliberately declines to distinguish 'agentic' from 'workflow' — both use LLMs; his concern is orthogonal: making whichever you use reliable and controllable.
• Gateway and gatekeeper
  His dual definition of code review's purpose: a gateway where the team shares information, learns and collectively owns the code, and a gatekeeper that governs and enforces quality at both code and process level.
• Viable coding / responsible vibe coding (Vibe-DD)
  The target state that keeps developer flow at the center (unlike TDD's correctness focus or SpecDD's behavior focus) but adds proactive rigor, correctness and ownership through always-on quality workflows — moving from 'vibe coding' to 'viable coding'.
• Agentic quality gate ('agentic CI/CD')
  Putting LLMs/agents into CI/CD quality gates — which he concedes is partly rebranded CI/CD, but argues the real opportunity is to break the one-way left-to-right flow so review and tests can apply to the plan, not just the code.
• Software DB
  His name for the combined RAG/context-engine plus rules/best-practices store in the reference architecture — the place where the system's learned standards and specs live.

Propositions

• explicitcontestedThe frontier of AI-for-development has advanced through successive ceilings — autocomplete, then agentic code generation, then code-review/quality workflows — and the next ceiling can only be broken by an adaptive, quality-grounded multi-agent system rather than by better single-agent code generation.conf: high
  supportsAccelerating Delivery: How AI Agents Can Own the Initial Code Draft · p5 — Adversarial multi-agent review producing standards-compliant code before a human sees it is a worked instance of Friedman's adaptive quality-grounded multi-agent system.
  contradictsGenie Lessons: Nobody Wants Agents · p1 — Beck: 'nobody wants agents or agent swarms.' Friedman champions multi-agent-with-quality as the path through the ceiling.
• explicitestablishedAI code generation roughly triples code output while producing about the same rate of defects per line, so it multiplies total issues and merely relocates the bottleneck from writing to reviewing/verifying — showing up as 'more code, not more end-to-end velocity'.conf: high
  supportsAgentic Code Review · p3 — Both find AI sharply raises raw code output while real value gain is small — Friedman's 'triple the code, same defect rate' mirrors Osmani's 4x-output / ~12%-value finding.
  supportsAgentic Code Review · p4 — Faros's churn-up / incidents-per-PR-up data corroborates Friedman's claim that AI multiplies issues and relocates the bottleneck to review.
  supports ←Five studies that are changing how I think about AI in software engineering · p4 — Demirer's attenuation of gains through the delivery hierarchy empirically confirms that AI 'merely relocates the bottleneck from writing to reviewing/verifying' — more code, not more end-to-end velocity.
  supports ←GitLab 19.2 Puts AI Agents to Work on the Security Backlog · p1 — Both describe AI code generation multiplying downstream review/security work rather than increasing end-to-end velocity — GitLab's paradox framing is the vendor-side mirror of this independently-sourced finding.
• explicitConsequently, the majority of investment in an AI-assisted SDLC should shift to verification (testing plus code review) and the context engine, not to code generation.conf: high
• explicitAI code-review agents are categorically different from code-generation agents: a code-gen agent treats your written rules probabilistically (following them only sometimes), whereas a well-built review agent treats each rule as a must-be-checked assertion — making review the correct place to enforce deterministic quality and compliance.conf: high
  supportsAgentic Code Review · p15 — Both locate reliable enforcement in a deterministic gate that cannot be talked out of its verdict — Friedman's review agent treats each rule as must-check vs the generator's probabilistic compliance.
• explicitA production multi-agent SDLC must include an arbiter that decides which agent is right when agents disagree (rival implementations, planner vs coder, tester vs coder), and the arbiter can only adjudicate if the plan/spec carries enough context — so investing in complete, machine-checkable specs is a precondition for arbitration.conf: high
  refinesGenie Lessons: Nobody Wants Agents · p2 — Friedman accepts that multi-agent adds a coordination/management burden (the 'who decides?' problem) and proposes the arbiter + human-observable comms as the mechanism to manage it, conceding it is not yet solved.
• explicitInter-agent communication must be made human-observable — recorded in surfaces a person can inspect, such as GitHub PRs and commits — rather than hidden inside opaque protocol channels (MCP/A2A/pub-sub/BigQuery logs), so humans retain oversight of what each agent did.conf: high
• explicitdevelopedGovernance, observability and controllability — including distinct least-privilege credentials per agent (e.g. the testing agent can reach staging, the coding agent cannot touch production) — must be a first-class layer of the architecture, not an afterthought.conf: high
  extends ←The Meter Was Always Running · p7 — That piece asserts governance, observability, and controllability (including per-agent least-privilege credentials) must be a first-class architectural layer; Haelen's signal table is a concrete design for exactly that layer.
• explicitThe context engine is no longer RAG but an agent in its own right (ingestion, queue and retrieval sub-agents) that fetches multi-source context, and context quality is the dominant determinant of trust — developers attribute about 88% of hallucination/distrust to context and rank improving it their number-one request; ~60% of Qodo's tool usage is context fetching, ~8% of it from non-code sources.conf: high
  supportsDesigning a Multi-Agent System for Engineering Support at Scale: A Case Study From Grab · p5 — Both name context management/retrieval as the dominant technical challenge for multi-agent systems; Friedman elaborates it as an agentic context engine.
• explicitcorroboratedHuman code review has quietly collapsed at AI volume — a large share of PRs merge with no human review (~80%) — driven both by review moving off-platform (only ~60% discuss reviews in GitHub/GitLab/Bitbucket) and by reviewers rationing attention inversely to diff size (5 lines get 50 comments, 500 lines get 'looks good to me').conf: high
  supportsAgentic Code Review · p6 — Same phenomenon: nobody decided to stop reviewing; at AI volume code merges unread and that became normal.
  supports ←AWS Expands DevOps Agent with AI-Powered Release Management to Validate Code Before Production · p2 — Matches the diagnosis that human review capacity has collapsed relative to AI-driven change volume (~80% of PRs merging unreviewed elsewhere in the corpus).
• explicitTests are executable specs, and having AI write tests improves the AI's code even when the tests are imperfect, because the token and reasoning spend on testing forces thinking that improves both artifacts — mirroring how writing tests helps a human reason about code.conf: medium
• explicitThe durable competitive edge is a system that continuously learns what quality means for your organization — your rules, best practices, standards and specs — so quality, not raw generation speed, becomes the differentiator.conf: high
  supportsAccelerating Delivery: How AI Agents Can Own the Initial Code Draft · p8 — Centralizing agent learnings into shared org intelligence is the mechanism behind Friedman's 'continuous learning of what quality means for you' as the moat.
  supportsGenie Lessons: Nobody Wants Agents · p3 — Agreement on outcome-orientation: Friedman's 'AI is the tool, not the solution — quality is the value' aligns with 'multi-agent is a feature; outcome-orientation is the value.'
• explicitThe goal is to move from 'vibe coding' (Vibe-DD, optimizing purely for flow) to 'viable / responsible vibe coding' that keeps flow at the center but adds proactive rigor, correctness and ownership through always-on quality workflows.conf: high
• explicitThe Google DeepMind multi-agent-system paper's 'no architectural advantage' finding does not invalidate multi-agent systems; its real contribution is the set of decisions you must make — how much can be parallelized independently, how many interfaces exist, whether you require an arbiter, and cost.conf: medium
• implicitA single AI agent cannot reliably both generate and judge its own work; separating generation from evaluation across distinct agents is what makes the quality workflow trustworthy.conf: medium
  supportsAgent Harness Engineering · p7 — Directly echoes 'separating generation from evaluation outperforms self-evaluation because agents skew positive grading their own work.'
  supportsAnthropic's Designs Three-Agent Harness Supports Long-Running Full-Stack AI Development · p2 — Same lever: separating the agent doing the work from the agent judging it improves output quality.
• implicitReliability must be reintroduced deterministically around non-deterministic generation at the verification/gate layer, because you cannot make the generator itself obey instructions reliably — reliability is a property of the wrapper, not of the model.conf: medium
  supportsSoftware Design in the Agentic Age: Place Your Bets · p11 — Shared thesis: let non-deterministic agents generate, but wrap them in deterministic verification (executable specs / must-check gates) that predictably finds faults.

Arguments

Narratives

• Friedman opens by asking the audience to raise hands for using one, two, three, four AI dev tools — counting Figma emitting code, Jira planning help, an SRE agent — then asks whether they will use more than three code-generating tools within a year.
  → Grounds the whole talk in the audience's lived reality that tool proliferation is already here, so coordination and control are the real problem.
• More than three cloud outages occurred in recent months; some of the affected companies say 10%, 30%, even 50% of their code is AI-generated — he explicitly disclaims a direct causal link but asks the audience to 'reflect on that'.
  → Motivates the reliability stakes of AI-generated code without overclaiming causation.
• Anthropic's published security-reviewer prompt essentially says 'from now on you are a senior security engineer, good luck', with a very long exclusion list that includes DDoS.
  → Illustrates naive single-agent 'vibe reviewing' and the inadequacy of prompt-only quality control.
• A Qodo survey asked whether Cursor follows written Markdown rules; no one answered 'completely followed'. The familiar experience: give a proper prompt with rules, run it seven times, get seven different implementations.
  → Evidence that generation is probabilistic about rules — the setup for deterministic 'must-check' review agents.
• A Qodo developer used Cursor's new web-app feature to change a ChatGPT-like app (move a button, edit text, remove a field) with no code written by hand — done well and quickly.
  → Concedes that code generation is genuinely magnificent, even on heavy-duty software.
• For that same 'simple' change, a computer-use testing agent running on staging caught that one of the chat options had actually been ruined — a regression in an element Cursor was not supposed to touch.
  → Shows autonomous/computer-use testing surfacing regressions that a quick human review would miss.
• Across millions of PRs, Qodo found 70% contained a high-severity issue, and a large share (~80%) had no human review — partly because only ~60% of teams discuss reviews on GitHub/GitLab/Bitbucket, partly the LGTM-by-diff-size meme.
  → Quantifies the collapse of human review and the resulting quality exposure, motivating AI review + arbitration.
• A Google DeepMind paper tested MAS architectures (centralized, decentralized, independent, hybrid) across financial and software scenarios and concluded there was 'no advantage'; Friedman jokes 'you just need one agent, prompt it differently — I'm kidding, at least partially', then mines the paper for its decision dimensions.
  → Pre-empts the strongest skeptic's evidence and converts it into his framework of considerations (parallelizability, interfaces, arbiter, cost).
• A tweet where 'Cursor didn't know what to do and Qodo butted in' — and the author asked ChatGPT who was right, Cursor or Qodo.
  → Dramatizes the unsolved arbitration problem: when two agents conflict, nothing yet decides.
• In a live product demo, a developer writes a ticket and the arbiter refuses to proceed because the ticket doesn't say which repo to change; the developer pastes the repo and the checkboxes turn green.
  → Makes concrete that arbitration depends on spec completeness — the arbiter first ensures there is enough information to adjudicate later.
• Some teams report that reaching 80% test coverage, which used to take three months, now takes one month — roughly 3x — after adopting agentic testing/review.
  → Evidence that incremental, bottleneck-first adoption of quality agents pays off.
• Q&A: asked how the survey was built, Friedman explains it combines three surveys (Qodo's own plus Sonar and Faros); they reached out to a variety of Qodo's 1,000+ clients, sought diversity and control, and stopped collecting at ~617 respondents once they saw enough diversity.
  → Discloses the provenance and sampling of the headline statistics — revealing both their basis and their vendor/self-selected limits.

Questions addressed

• ✅ Why does AI code generation not translate into end-to-end delivery velocity?
  Because it triples code while defect-rate-per-line holds, so it multiplies issues and moves the bottleneck to review and verification rather than removing it.
• ✅ If AI writes both code and tests, how can AI-written tests help — wouldn't a wrong model be wrong in both?
  The token and reasoning spend on testing forces thinking that improves both artifacts, just as writing tests helps a human reason, even when individual tests fail.
• ✅ Are multi-agent systems worthless, as the DeepMind 'no advantage' paper suggests?
  No; the paper's finding is reframed as a set of design decisions (parallelizability, number of interfaces, need for an arbiter, cost) rather than a refutation — though he never actually engages the finding that architecture choice showed no advantage.
• ⭕ Who decides when two agents disagree?
  He prescribes a designated arbiter that needs a sufficiently complete spec, but explicitly calls it 'one of the hardest things' and 'not solving the problem yet' — a direction, not a solution.
• ✅ How should agents communicate — via protocols (MCP/A2A) or traditional pub/sub?
  Use whatever mechanism, but design it so the exchange is recorded in a human-observable UI (e.g. GitHub PRs/commits) rather than opaque channels.
• ✅ Is the 'agentic quality gate' just rebranded CI/CD?
  Partly yes, but the opportunity is to break the one-way left-to-right CI/CD flow so review and tests can apply to the plan, not only the code.
• ⭕ How do you mature AI adoption from small companies (where it dominates) to the enterprise?
  Raised as the key question but answered only indirectly — via quality workflows, governance and incremental bottleneck-first adoption — without an enterprise-specific playbook.

Gaps

• incompleteThe arbiter is asserted as necessary but its decision procedure is never specified: when the planning agent and coding agent (or two rival implementations) conflict, how does the arbiter actually choose — confidence scores, voting, running the tests, escalation to a human? Beyond 'have a complete enough spec' and 'have enough context', the adjudication mechanism is a blank.
  Read next: Read multi-agent orchestration and LLM-judge/evaluator literature (e.g. Anthropic's three-agent harness and its 'what if the evaluator is wrong' failure mode) and consensus/adversarial-review patterns.
• incompleteNearly every quantitative claim (2x code quality, 47% more productivity from AI review, 70% of PRs high-severity, ~80% no human review, 88% context-attribution) comes from vendor-run or vendor-adjacent surveys — Qodo's own plus Sonar and Faros — sampled from 'at least half of our [1,000+] clients', i.e. self-selected AI-tool users, n≈617, with no independent baseline or controlled non-AI comparison. The figures are correlational and commercially interested, not causal measurement.
  Read next: Read independent longitudinal/controlled measurement (METR's randomized study, full Faros methodology, DORA) and treat the vendor figures as directional, not established.
• incompleteThe rebuttal of the DeepMind 'multi-agent shows no advantage' paper is promised and then sidestepped: he reframes it into his four considerations without ever engaging the actual finding that architecture choice (centralized/decentralized/independent/hybrid) produced no advantage. Whether his adaptive quality-grounded MAS would beat a single well-prompted sequential agent is left unproven.
  Read next: Read the DeepMind MAS paper directly and the multi-agent-skeptic case (Beck's 'nobody wants agents') to judge the claim on its own evidence.
• incompleteCost/economics of running the full parallel quality stack (multiple testing, review, context and arbiter agents on every change) is named as 'very important' and then explicitly deferred to last and never quantified — no token-cost figure, no comparison of the multi-agent quality stack's cost against the productivity it recovers.
  Read next: Read AI inference unit-economics and the token-cost gaps flagged for adversarial multi-agent review workflows.
• incomplete'Continuous learning' of your quality standards/specs is asserted as the moat, but the mechanism is unspecified: who curates the learned rules and best practices, how is drift or poisoning of the learned standards prevented, and how does the system avoid ossifying bad conventions it observed in old PRs?
  Read next: Read on knowledge curation for agent memory and organizational-legibility of standards (who owns the rule set, how it is reviewed and versioned).

Closes gaps elsewhere

• Genie Lessons: Nobody Wants Agents · g1 — Provides a worked example of the multi-agent-as-internal-architecture reading that Beck never separated from the user-facing swarm model: planner / coder / tester / reviewer / context agents orchestrated behind the scenes, with their communication deliberately surfaced to a human via a UI layer (recorded in GitHub PRs and commits). It demonstrates the distinction rather than engaging Beck's critique.

Related documents

• Genie Lessons: Nobody Wants Agents
• Software Design in the Agentic Age: Place Your Bets
• Anthropic's Designs Three-Agent Harness Supports Long-Running Full-Stack AI Development
• Agent Harness Engineering
• The Anatomy of an Agent Harness
  Context Engineering Evals Guardrails Multi-Agent Orchestration
• The New SDLC With Vibe Coding: From ad-hoc prompting to Agentic Engineering
  Code Was Never the Bottleneck Context Engineering Evals Guardrails
• Agentic AI and Code Reviews
  Context Engineering Multi-Agent Orchestration Pull Requests & Async Code Review
• Article: Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules
  Agent Governance (Internal) Evals Guardrails
• Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation
  Evals Guardrails Multi-Agent Orchestration
• Inside Claude Code Auto Mode: Anthropic's Autonomous Coding System with Human Approval Gates
  Agent Governance (Internal) Guardrails Multi-Agent Orchestration