Presentation: From Retrieval to Reasoning: Building Production-Ready Agentic AI Systems with Knowledge Graphs

Cassie Shum · rss · 2026-09-12 source ↗ Reader 🖥️

practical · conference talk transcript with live demo and audience Q&A — QCon AI presentation; personal-lessons-learned framing, explicitly disclaimed as neither dogma, a GraphRAG tutorial, nor a framework prediction. Two parts: why a knowledge graph is the foundation of agentic systems, then four practical patterns demonstrated via an internal engineering harness.

Structure

Key terms

• context bundle / context as a bundle
  A task-specific assembly of graph-linked sources — domain knowledge, the current spec, prior related work, engineering standards, and the relevant code repo(s) — selected by proximity in the knowledge graph (e.g., which policy documents are in 'my team's neighborhood') rather than by dumping every available document into the prompt.
• provenance (for decision making) Software Supply-Chain Security
  A queryable, branching audit trail in the graph linking an idea, to spec, to every human-agent back-and-forth intervention, to the merged PR, to production — captured continuously as a 'flywheel' where each action is written to the graph and then summarized.
• code as truth, spec as kernel
  The implementation, not the specification, is what actually is real and driven to production; the spec's lasting value is capturing intent, and the two must be reconciled and related to each other in the graph rather than the spec being treated as disposable and re-derived from scratch each pass.
• visibility (agent/team visibility) Organizational Legibility
  Retrospectively queryable records of agent activity — tokens used, turns taken, cost per task, context bundle contents and size, tasks shipped over time — surfaced from the same graph that already holds context and provenance data, used for audits, reporting, and team-effectiveness review, not merely a live progress display for one user.
• knowledge graph as substrate (vs. retrieval system)
  The graph functioning as the foundational infrastructure layer of the engineering practice itself — feedback-loop tracking, audit, visualization — rather than chiefly a mechanism for producing a fuller answer to a question.
• fast, honest mirror
  Agents don't independently improve or degrade outcomes; they accelerate whatever engineering practices (good or bad) already exist in an organization, exposing weak practices as fast, unmaintainable output and strong practices as fast, reliable delivery.

Propositions

• explicitcontestedKnowledge graphs are shifting from being valued for retrieval accuracy (an edge eroded as models got better at parsing raw documents) toward serving as a substrate for engineering practice: context assembly, decision provenance, spec/code reconciliation, and visibility.conf: high
  extendsComprehension at AI Speed: Building a Context Store for Evolutionary Architecture · p5 — Provides an independent, knowledge-graph-based instantiation of the 'context store as substrate, not retrieval optimization' idea, with an empirical (though small) token/turn efficiency finding the context-store piece asserts only conceptually.
  contradicts ←Graphify: Unifying Codebase Context to Streamline Agentic Software Engineering · p8 — Graphify's own value proposition is framed almost entirely around retrieval/token-efficiency ('token reductions compared to naive file reading') — precisely the framing Shum's talk argues has eroded as a durable differentiator now that models parse raw context well. The community's own report that grep still feels faster on mid-sized repos is empirical evidence for exactly the skepticism Shum voices about retrieval-accuracy-first graph tools.
• explicitA knowledge-graph-backed organizational context layer lets agents answer org-specific questions (which compliance regime applies, who owns a policy, what was decided last quarter) with much higher fidelity than an agent relying on general model knowledge or an internet search.conf: high
  supportsWhy AI agents need a Context Lake · p4 — Both argue a centralized organizational knowledge layer is what lets agents give consistent, org-specific (vs. generic) answers.
  supportsWhy AI agents need a Context Lake · p2 — Matches the specific failure mode: an agent with tool/API access still cannot answer basic organizational questions (who owns this, which regime applies) without a relational knowledge layer.
• explicitAgents are a "fast, honest mirror" of an organization's existing engineering culture: they accelerate good practices into fast, reliable delivery, and just as quickly turn undefined requirements, brittle handoffs, and inconsistent data into unmaintainable code ("AI slop").conf: high
  supportsPresentation: AI-First Software Delivery: Balancing Innovation with Proven Practices · p5 — Near-identical claim reached independently: AI amplifies existing engineering practices, whether strong or weak.
• explicitAgentic tooling collapses the feedback loop between writing an under-specified spec/story and discovering it was wrong — from roughly two to four weeks (or, historically, six months) down to about 40 minutes.conf: high
  supportsSam Hatoum's Post · p3 — Both describe agentic tooling collapsing the spec-to-discovery feedback loop from months/weeks to minutes.
• explicitIn a controlled internal experiment (on-call policy documents across three teams), giving an agent a knowledge-graph-curated context bundle produced output identical in correctness to giving it every available document unfiltered, but used markedly fewer tokens and fewer agent turns to reach that answer.conf: high
• implicitTracking decision provenance (idea -> spec -> human/agent back-and-forth -> merged PR -> production) is valuable specifically because real decision paths are not linear — "it will take a hop here, it will take a hop here" — so a flat commit history or ticket system does not capture the branching, iterative path by which a decision actually formed.conf: medium
• explicitRegenerating the specification from scratch at each work pass destroys the most valuable information — what the human and agent discovered together while implementing — so a summarized reconciliation between the original spec-intent and the actual code-reality should be written back into the shared knowledge graph rather than the spec being treated as disposable and re-derived.conf: high
  supportsWhen AI Writes the Code, Specifications Need an Exit Strategy · p7 — Both observe that discovery continues throughout implementation and must be fed back into the working spec/context rather than treated as complete at the start.
• explicitCode, not the specification, is what is actually real and driven to production, while the spec's lasting value is capturing intent, not dictating final behavior.conf: high
  supportsWhen AI Writes the Code, Specifications Need an Exit Strategy · p3 — Shum reaches the same code-as-truth position from practitioner experience (code is what's real, driven to production); Eisele independently supplies the stronger de-facto-contract argument (an unintended behavior running unchanged for years becomes a binding contract) that Shum's talk does not itself make.
  refinesWe've Stopped Arguing About Whether AI Changes Software · p4 — That extraction leaves open whether code remains 'the artifact that matters' or becomes an ephemeral byproduct regenerated from spec; Shum's practical stance answers this from one side, with a mechanism (graph-based intent/reality reconciliation) for keeping both without discarding either.
• explicitTreating an agent as "fire and forget" — handing it a spec and waiting for finished output without a continuous human-agent feedback loop — fails to produce what was actually wanted.conf: high
• implicitWhich underlying agent/model executes a task is not fully interchangeable in practice, because context learned from prior corrections and successful patterns needs to be deliberately carried forward into the next feature, and simply swapping engines risks losing that accumulated, task-specific learning.conf: medium
• explicitVisibility into agent activity — tokens used, cost per task, context bundle size, number of tasks shipped over a period — should be captured so it can be queried retrospectively (audits, reports, team-effectiveness reviews), not just observed live while an agent runs.conf: high
  supportsThe Meter Was Always Running · p3 — Both treat the turn — not the request or the final answer — as the unit that visibility/observability must be built around: Shum's harness counts turns per task for cost/efficiency reporting, matching the claim that agent observability must be architected around the turn as the unit of observation.
• explicitdevelopedA knowledge graph carries real build-and-maintenance overhead, and adopting one is only justified when a demonstrated benefit (organization-specific context, or turn/token efficiency at scale) outweighs that cost — it is not a default architectural choice.conf: high
  refinesTony Seale's Post · p11 — Vault-LD asserts the graph/ontology overhead is generally worth its payoff; Shum adds the conditionality that it is only worth it when a specific, demonstrated benefit (org-specific context, efficiency at scale) is present.
  refines ←Enterprise Analytics Beyond Dashboards: Intelligent Data Orchestration with LLMs · p18 — Khapekar frames early knowledge-graph investment as unconditionally high-leverage ('invest early... highest-leverage'); the other document is more conservative, treating graph adoption as justified only once a demonstrated benefit exists that can't be gotten more cheaply elsewhere — together they bound the investment-timing question this document leaves internally unresolved (see its own argument-strength tension between 'invest early' and 'pick one use case and go deep').
• implicitA knowledge graph's performance overhead is a function of graph/data scale and traversal design rather than an intrinsic property of "using a knowledge graph" — an architecture that adds negligible overhead at small scale (~30 documents, one team) may behave very differently at organization scale.conf: medium
• explicitdevelopedInstitutional/tribal knowledge — informal rules held partly in engineers' heads and partly scattered across stale, contradictory wiki pages — is largely inaccessible and confusing to agents unless it has been deliberately captured and structured, because agents have no principled way to reconcile which of several outdated or conflicting sources still applies.conf: high
  supportsThree Pillars of Platform Engineering: A Virtuous Cycle · p5 — Both identify tribal knowledge (held in engineers' heads, undocumented) as a genuine scaling problem rather than a minor inconvenience.
  supportsAI productivity is burning out your best engineers · p3 — Both note that AI/agent tooling is not designed to account for undocumented tribal knowledge, leaving a structural gap rather than an individual failing.
  extends ←Hear me out: Agentic TPMs (Technical Program Managers) · p5 — Proposes a concrete practical mechanism (an agent in a role whose job is explicitly to surface and document it) for exactly the tribal/institutional-knowledge-inaccessibility problem that proposition names, though without addressing that proposition's implicit reconciliation challenge.
• implicitBecause organizational rules and decisions change over time (what was correct six months or five years ago may no longer be), a usable knowledge-graph context system needs to track a decision's lineage and currency — when it was made, what superseded it — or agents will retrieve and apply rules that are no longer valid as if they still were.conf: medium

Arguments

Narratives

• Cassie's own career arc — from coder, to a "hand-wavy" architecture consultant at Thoughtworks and elsewhere, back to the codebases via agentic engineering, pivoting her graph practice based on lessons from the last six months.
  → Establishes personal authority for the practical, lessons-learned framing and licenses the talk's explicit epistemic humility ("this is not dogma... just Cassie's experience").
• The Auth0/OAuth demo — an idea ("implement Auth Auth") bound in the harness to an internal architecture Google Doc and a GitHub repo, so the agent works from the organization's own prior implementation and spec instead of researching Auth0/Stripe/Facebook patterns generically.
  → Makes the abstract "context bundle" pattern concrete; illustrates and supports the claim that org-specific context beats generic model/internet knowledge.
• The on-call-policy A/B experiment across teams Meridian, Squad Cypress, and Atlas — the same task run twice, once with a graph-curated 'nearest neighbor' set of policy documents, once with all ~30 documents dumped in unfiltered.
  → Serves as the talk's sole quantitative evidence; grounds the pivot away from "GraphRAG for accuracy" toward "knowledge graph for token/turn efficiency."
• Her partner's startup co-founder vibe-coded an entire application in three months, demoed it successfully to an excited customer, then everything fell apart once it needed to go to production and scale, for lack of testing and CI/CD discipline.
  → Illustrates and supports the "fast, honest mirror" thesis — agents accelerate whatever engineering practice (or its absence) already exists.
• Her own prior claim, made on a podcast six months earlier, that "code is cheap, spec is the real truth" — which she now calls naive.
  → A self-corrective narrative that legitimizes Pattern 3 (code as truth) as a hard-won, revised lesson rather than an armchair claim.

Questions addressed

• ✅ Is GraphRAG dead — does a knowledge graph still add value now that models are so good at parsing raw context?
  Not dead, but its value proposition has moved from retrieval accuracy to efficiency (tokens/turns) plus non-retrieval functions (provenance, code-truth reconciliation, visibility).
• ✅ How do you keep specs from going stale or wrong as agents implement quickly?
  Treat code as the authoritative truth, keep the spec brief and intent-focused, and continuously reconcile spec-intent vs. code-reality back into the shared graph rather than regenerating the spec from scratch each pass.
• ⭕ How do organizational/tribal knowledge and shifting institutional rules stay usable by agents over time?
  She names the problem clearly (stale wiki pages, dead rules, decisions superseded over time) and asserts the graph should capture decision lineage, but describes no working mechanism for automatically detecting when a captured rule or decision has been superseded.
• ✅ When is a knowledge graph worth its overhead versus simpler localized-agent-with-links approaches?
  Use it only when a demonstrated benefit exists (organization-specific context needs, or efficiency at scale) — it is not a default choice given its real build/maintenance cost.
• ⭕ Is the harness's own audit trail (the agent writing its actions and summaries back into the graph) trustworthy enough for governance/audit purposes, given the agent both performs and reports on its own work?

Gaps

• incompleteThe harness's provenance/audit trail is written back into the graph by the same agent (or an agent-driven summarization step) that performed the work, with no described independent, agent-can't-edit instrumentation layer verifying the record matches what actually happened — a self-reported chain of custody.
  Read next: Read against extract-the-meter-was-always-running (especially its claim that an agent cannot be the system of record for its own authority, cost, or refusals, and that evidence must be captured at instrumentation points the agent cannot bypass) for a corpus-level argument that self-reported agent provenance specifically fails an auditor.
• incompleteThe core empirical claim — that a knowledge-graph-curated context bundle reduces tokens/turns versus dumping all documents — rests on a single internal experiment across roughly 30 documents in one organization for one task type (on-call policy), explicitly not yet validated for accuracy or savings at customer scale.
  Read next: Look for a follow-up talk or post from Cassie Shum / RelationalAI reporting the promised customer-scale validation, or independent benchmarks comparing graph-curated context assembly against raw-document retrieval on token/turn efficiency.
• uninformedThe talk never engages with non-knowledge-graph approaches to the same four problems — e.g. a plain versioned document store with metadata tagging for context bundling, git/PR history for provenance, or a dedicated telemetry platform for visibility — so it is unclear whether a knowledge graph specifically is necessary or simply one implementation choice among several.
  Read next: Compare against extract-context-store-evolutionary-architecture's 'context store' proposal (a deterministic, versioned, repo-native record of intent, behavior, and architectural conformance) that claims similar benefits without a graph database — do the two proposals differ in capability, or mainly in storage technology?
• incompletePatterns 2 and 4 (provenance and visibility) both assume that once everything is 'in the graph,' useful queries 'come for free,' but the talk gives no account of who designs and maintains these queries, how results stay honest as the graph schema evolves, or what happens when the harness itself — by her own admission in Q&A 'a bunch of devs clamoring something together' rather than a maintained product — breaks or drifts.
  Read next: Look for material on internal-tooling sustainability and platform-as-product discipline to test whether an informal internal harness like this one survives team turnover or organizational scaling.
• incompleteThe claim that 'domain and context changes over time' and that decision lineage needs tracking is asserted as important, but no mechanism is described for how the graph detects that a previously captured rule or decision has been superseded, versus simply accumulating an ever-growing, potentially contradictory set of historical facts that an agent must disambiguate among — recreating, inside the graph, the very 'which policy applies' problem a plain agent already has.
  Read next: She flags this herself in the Open Questions section ('agentic memory architecture... how do we do this at scale'); worth tracking whether a later Shum/RelationalAI publication addresses temporal invalidation of graph facts.
• illogicalThe claim that 'the moat isn't the models... it's the domain knowledge and the graph you put together' sits in tension with her own Q&A answer that the graph's original differentiator (retrieval accuracy) has been erased by model improvements, and that a graph should only be built when a specific benefit justifies its overhead. If the moat is the domain knowledge itself — which her own Pattern 1 experiment shows a sufficiently capable model can already use well from plain documents — it is unclear whether graph structure per se is the moat, or merely one delivery mechanism for domain knowledge that models will keep needing less structural help to use.
  Read next: Watch for this tension against extract-context-store-evolutionary-architecture and other retrieval-augmented-generation corpus material on whether structuring context in a graph remains a durable moat as base-model context handling keeps improving.

Related documents

• The Meter Was Always Running
• AI productivity is burning out your best engineers
• Three Pillars of Platform Engineering: A Virtuous Cycle
• Sam Hatoum's Post
• Presentation: AI-First Software Delivery: Balancing Innovation with Proven Practices
• Comprehension at AI Speed: Building a Context Store for Evolutionary Architecture
• Coding Agent Development Workflows
  Architecture Decision Records Context Engineering Organizational Legibility Spec-Driven Development
• Why AI agents need a Context Lake
  Context Engineering Knowledge Base Organizational Legibility
• Graphify: Unifying Codebase Context to Streamline Agentic Software Engineering
  Context Engineering Knowledge Base Retrieval & RAG
• No Vibes Allowed: Solving Hard Problems in Complex Codebases – Dex Horthy, HumanLayer
  Context Engineering Organizational Legibility Spec-Driven Development