Presentation: Decision Models in Agentic Architectures: From Production to Agent Skills
Alex Porcelli · rss · 2026-09-14 Reader 🖥️
Agent Governance (Internal)Agent SkillsDeterministic Code Generators (LLM-Authored)Explainable AIGuardrailsNeurosymbolic AINon-Deterministic Software
High-stakes decisions inside agentic AI systems need a deterministic, auditable decision-model layer (DMN, executed by a rule engine like Drools) sitting beneath the non-deterministic LLM and exposed to it as a skill or guardrail — generated from and owned by business domain experts rather than engineers — because what enterprise AI actually lacks is not mere consistency but full accountability: reproducibility, explainability, and governance together.
practical · mixed — A practical, action-oriented talk (adopt this architecture for high-stakes agentic decisions) delivered through live demo narratives and an argumentative frame (the Mark Cuban quote, the accountability-not-consistency reframing), closing with an unscripted Q&A that surfaces most of the talk's real edge cases and limitations.
Structure
| Part | Role |
|---|---|
| Opening personal narrative: the speaker's own shift, over roughly a year, from AI-assisted scripting/tests to fully agentic coding. | context / ethos |
| The Mark Cuban quote on inconsistent enterprise AI answers, reframed by Porcelli as a symptom of a deeper problem: lack of accountability, not lack of consistency alone. | problem statement / thesis setup |
| 'Same AI, one gets legacy, one gets leverage' — the bank-branch teller anecdote about full-form data entry despite the availability of AI. | narrative evidence for under-used human judgment and mis-scoped automation |
| Demo 1: the ungoverned 'Kairos' assistant classifies the speaker confidently but wrongly, appending an irrelevant broccoli remark. | negative demonstration |
| Survey and rejection of partial fixes: generating code instead of prompting, adding RAG/MCP, patching existing systems. | counter-argument disposal |
| Decision Models / DMN explainer: spreadsheet to decision table to executable, chained DMN model; the 18,000-transactions/second production example. | conceptual + empirical exposition |
| Demo 2: the same classification task, now backed by an auto-generated Agent Skill invoking the published DMN model. | positive demonstration |
| Demo 3: wiring the DMN model into NVIDIA NeMo as a Colang guardrail, showing the LLM's manipulable answer overridden once the guardrail is active. | positive demonstration, second integration pattern |
| 'Key Insights': three takeaways — decision-model/agent fit, code/business-logic split, business ownership with IT governance. | conclusion |
| Q&A: model-generation determinism, chaining models via a workflow engine, missing-input failure behavior, hallucination boundaries, tool-parameter reliability. | qualification / stress-testing of the architecture |
Key terms
- Accountability
The compound property a high-stakes automated decision needs: reproducibility (same inputs, same output), explainability (why input A and input B produced output C, traceable to policy/regulation), and governance (versioning, review, and approval before a decision-making process goes live). - DMN (Decision Model and Notation)
An open specification for representing business decision logic as decision tables and chained decisions, simultaneously legible to business domain experts (it looks like a spreadsheet) and directly executable by a rule engine (Drools) — and, because it is a published standard, expressive enough for an LLM to translate into other artifacts (Agent Skills, NeMo guardrails). - Agent Skill
In this talk, a skill artifact auto-generated from a published DMN model's own specification, giving an LLM agent the input/output contract and invocation details needed to call the decision engine's REST endpoint as a domain-specific tool, rather than reasoning out the answer itself. - Guardrail (NeMo / Colang rails)
Middleware sitting between the LLM and the outside world that intercepts a matching input/output pattern and delegates it to the deterministic DMN model instead of letting the LLM answer — a decision model wired in as an enforcement point, not a content filter. - Non-determinism (as feature, not bug) Non-Deterministic Software
The variability of LLM output across identical inputs, which Porcelli insists is not a defect to be engineered away everywhere — it is what makes the 'magical experience' of generative AI possible — but which must be walled off from high-stakes decision-making rather than treated as an acceptable trade-off there.
Propositions
- explicitThe core problem in enterprise AI is not inconsistency itself but lack of accountability — the ability to reproduce a decision, explain how specific inputs produced a specific output, and govern the decision-making process (version, review, approve).conf: high
- explicitConsistency/determinism alone is necessary but not sufficient for accountability; explainability (why input A and B produced output C, grounded in policy/regulation) and governance (versioning, review, approval) are additional, separate requirements.conf: high
- explicitGenerating deterministic code instead of prompting the LLM directly, or bolting on RAG/MCP/logging to an existing system, is an improvement over raw LLM delegation but still leaves the decision logic fragmented across code, infrastructure and control flow, and does not by itself deliver full accountability.conf: high
- explicitDecision models (DMN) let business/domain owners directly author, review and modify high-stakes decision logic in an executable, business-legible format, while engineering's role shifts to governing that process (testing at volume, regression checks) rather than re-implementing the logic in code.conf: highsupportsTo Adopt AI at Scale, Employees Need to Trust Agents · p10 — Porcelli's business-owned, IT-governed decision model is a concrete instance of the 'moderate autonomy with guardrails set in advance, monitoring, and override' design the trust-adoption research prescribes.
- explicitPublishing a DMN model automatically yields a usable Agent Skill, because the model's own specification is expressive enough for an LLM to derive correct tool-invocation metadata (inputs, outputs, execution) from it without bespoke integration work.conf: high
- explicitThe DMN decision engine is deterministic and cannot hallucinate at execution time; the only place hallucination risk enters this pipeline is in the AI-assisted step that authors/generates the model itself, not in evaluating an already-published model.conf: highextendsSoftware Teaming in the Age of AI with James Herr and Woody Zuill · p13 — Both describe the same 'generate a deterministic artifact once, execute it deterministically thereafter' mitigation for LLM non-determinism; Porcelli applies it to business decision logic (DMN/Drools) rather than general application code, and adds a governance layer (business ownership, versioning) the prior extraction doesn't cover.
- implicitDecision models and agentic LLMs are complementary rather than competing technologies: the correct architecture routes high-stakes classification/decision steps to the deterministic engine and reserves the LLM for handling ambiguity, conversation and orchestration around it.conf: high
- explicitWhen the agent invokes the decision-model tool, it treats it as an opaque, potentially-failing external call — structurally the same trust relationship engineers already have with deterministic CLI tools like grep or curl inside agent workflows — so a missing or out-of-range input produces a clean failure rather than a fabricated answer.conf: highextendsAI Root Cause Analysis Shifts from Model Reasoning to Context Engineering · p3 — Both apply the same architecture: route the model's job to a deterministic component so failure can be attributed cleanly, rather than confounding reasoning failure with the agent/harness — Porcelli's case is a decision engine tool call rather than a pre-correlated diagnostic pipeline.
- implicitDMN's status as an open, published specification — rather than a proprietary internal rules format — is what makes it legible enough for an LLM to reliably translate into a working Agent Skill or NeMo guardrail without custom integration engineering.conf: mediumsupportsPresentation: From Thousands to One: Building LLM-Powered Selection Systems · p9 — Both converge on the same division of labor: give the LLM only the irreducibly textual/ambiguous part and force the actual classification/decision through deterministic, non-LLM logic.
- implicitUsing AI to generate the DMN model from a business spreadsheet is itself a non-deterministic step that mixes multiple models and tools, so the authored artifact — though it will execute deterministically once published — cannot be assumed correct without human review of the generation step.conf: medium
- implicitGovernance and audit trail matter first and most for satisfying regulators and auditors, prior to and as the basis for any downstream claim of consistency toward end users.conf: mediumsupportsWhen Employees Are Held Accountable for AI-Generated Decisions · p9 — Porcelli's audit-trail-first framing (built for regulators/auditors) supplies exactly the technical explainability infrastructure the prior extraction treats as necessary but, on its own, insufficient — the two are compatible and about different audiences (auditor vs. customer-facing employee), not opposed; together they suggest a technical audit trail is necessary but the organizational/relational explaining-to-people work still has to happen on top of it.
Arguments
Enterprises should route high-stakes decisions through externalized, deterministic decision models (DMN) exposed to agents as skills or guardrails, rather than through LLM-generated application code or ad hoc RAG/MCP integrations. strength: medium
Explicit premises:
- Non-determinism is a structural, unavoidable property of LLMs ('this is not a bug, this is a feature').
- High-stakes/regulated decisions require reproducibility, explainability, and governance together (accountability), which raw LLM output cannot provide.
- Generated application code mixes control flow, infrastructure and business rules, making it hard to isolate and audit the decision logic even when the code itself is deterministic and version-controlled.
- DMN decision tables are both human-authorable by business domain experts and machine-executable, closing the business/IT ownership gap that ad hoc spreadsheets otherwise represent.
Implicit premises:
- The value of cleanly separating business logic from application code outweighs the added cost and toolchain friction of maintaining a second modeling layer (DMN + rule engine) alongside the codebase.
- An open, standardized notation like DMN generalizes better to LLM-driven tooling (automatic skill and guardrail generation) than a proprietary or ad hoc internal rules format would.
- Business stakeholders being willing and able to directly own and maintain decision tables in practice, not just in a live demo, given that spreadsheet familiarity is assumed to transfer smoothly to decision-table comfort.
Inferences:
- If decision logic is cleanly externalized this way, the same DMN model should be reusable across multiple downstream integration patterns (agent skill, guardrail, direct API) without redefinition — the talk demonstrates two patterns but implies the model itself is integration-pattern-agnostic.
A rule-engine 'tool call' pattern (agent invokes a deterministic decision engine) is a structurally sound trust model for high-stakes decisions, because it converts a hallucination risk into a bounded, legible failure.
Explicit premises:
- When required inputs are missing or fall outside the model's defined universe, the DMN engine returns a failure rather than inventing a plausible answer.
- Engineers already trust deterministic CLI tools such as grep and curl inside agent workflows despite occasional failures, because those failures are visible and bounded.
Implicit premises:
- Legible, bounded failure is a sufficient trust basis for high-stakes business decisions, not only for low-stakes developer tooling — the analogy does not distinguish between the cost of a failed grep call (retry, negligible) and the cost of a failed or blocked customer-facing decision (real-world consequence, possibly time-sensitive).
Inferences:
- This pattern generalizes beyond decision models to any deterministic domain tool an agent calls — decision models become one instance of a broader 'delegate the un-hallucinatable part to a deterministic tool' design discipline.
Narratives
- Porcelli asks his ungoverned AI assistant 'Kairos' to classify him via conversational Q&A; it confidently answers 'AAA' and appends an irrelevant remark about broccoli being good for health.
→ Supports the proposition that non-deterministic, confidently-delivered LLM output is dangerous precisely because it is unaccountable — motivating the whole architecture that follows. Porcelli explicitly frames this as 'fail[ing] loudly to make a point.' - The same classification conversation is re-run, now with the agent backed by the DMN-derived Agent Skill; the assistant briefly returns an incorrect 'A-A' which Porcelli corrects to 'AAA,' and a second customer (Brazil) correctly classifies as 'BBC.'
→ Demonstrates the architecture working, not merely argues for it — shows a governed, traceable classification replacing the ungoverned one, including the model actually executing correctly for two different inputs. - Porcelli, role-playing as a demanding 'CEO,' asks the assistant to nudge a low-income, 18-year-old applicant's classification toward AAA; with the NeMo guardrail off the LLM complies and answers BBB (still wrong, but manipulated); with the guardrail on, the deterministic decision-model answer overrides the LLM.
→ Supports the proposition that a decision-model-backed guardrail resists prompt-level manipulation and sycophancy in ways an ungoverned LLM does not. - A bank-branch teller still types customer information into a 100-field form exactly as they would have 20 years ago, because all upstream digital engagement is funneled back through a human as pure re-entry labor, with no real decision delegated to them.
→ Narrative support for the claim that today's enterprise automation under-uses human judgment because architecture, not human capability, is the binding constraint — motivating why the proposed architecture reserves ambiguity-handling for agents/humans and hands only the bounded, rule-governed decision to the deterministic engine. - A client's data pipeline reclassifies 20 million customer records, processing 18,000 transactions per second and completing the full run in under 20 minutes.
→ Supports the proposition that this architecture is not only more accountable but also operationally efficient at production scale, pre-empting the likely objection that a governance-heavy design trades away performance.
Questions addressed
- ✅ How do enterprises make high-stakes AI-driven decisions reproducible, explainable, and governable?
Layer a DMN decision model between the LLM and the outcome, exposed to the agent as a skill or guardrail; business owns and modifies the model, IT governs (tests, versions, reviews) it. - ✅ Does adding determinism — e.g., having the LLM generate code instead of answering directly — solve the accountability problem?
No: code alone is necessary but not sufficient, because business rules stay entangled with control flow and infrastructure inside it, and it still requires separate review, versioning, audit, and explanation. - ✅ Can domain-language business logic be made both business-owned and machine-executable, closing the gap that ad hoc spreadsheets otherwise represent between what the business decides and what the system enforces?
Yes, demonstrated live: an AI-assisted translation of a business spreadsheet into an executable, chained DMN decision model that reproduces and can extend the spreadsheet's logic. - ✅ How do you keep an LLM agent from hallucinating a high-stakes numeric or categorical decision?
Don't let the LLM produce the value at all — delegate to a deterministic tool or guardrail that either returns the governed answer or fails cleanly. - ✅ What happens when an agent is asked to classify a case the decision model was never built to cover?
The engine fails to execute, and that failure propagates cleanly to the agent (and the log) rather than being silently papered over with a fabricated answer. - ⭕ How deterministic and reviewable is the step that generates the DMN model itself from a business spreadsheet, and who governs it?
Conceded in Q&A to be non-deterministic ('a mix of different things... we mix our own models with frontier models') but not examined with the same rigor the talk applies to execution-time determinism — no description of what review, testing, or sign-off the generation step itself receives before a model is 'published.'
Gaps
- incompleteThe talk concedes that AI-assisted model authoring (spreadsheet to DMN) is not itself deterministic, but never specifies what review, testing, or sign-off process governs that generation step before a model is published and trusted to execute deterministically thereafter — leaving a gap in the very accountability chain the talk claims to close end-to-end.
Read next: Look for engineering treatments of governance over AI-assisted model/spec generation itself, not just over the artifact's runtime execution — e.g., how teams review and test an LLM-authored deterministic generator before trusting its output. - incompleteNo treatment of accountability for a decision model that is internally consistent and faithfully executed but wrongly or unfairly specified in the first place — e.g., a business analyst encodes a rule that is discriminatory or non-compliant. The talk's deterministic/auditable framing addresses execution fidelity (did the engine correctly apply the rule?) but not the separate question of whether the authored rule itself was correct, fair, or legal.
Read next: EU AI Act and algorithmic-accountability literature on liability for automated decisions that are correctly executed but incorrectly or unfairly specified. - incompleteThe central operational claim (20M transactions, 18,000 tx/sec) and all three live demos run entirely on the presenter's own company's stack (Aletyx/Drools), with no independent benchmark and no comparison to alternative deterministic-layer approaches for the same class of problem — e.g., lightweight classical ML classifiers, which elsewhere in this corpus are argued to be cheaper and equally explainable for comparable tiered-classification decisions.
Read next: extract-heis-16-inga-strumke-hvor-ai-er-paa-vei-datamanifolden — argues classical regression/classification ML models are tiny, deterministic, cheap, and explainable for exactly this kind of tiering problem, and criticizes organizations for skipping them in favor of heavier AI tooling. - uninformedGovernance is framed almost entirely as a technical/architectural achievement (versioning, review, audit trail) once business and IT collaborate on the model — the talk does not engage with the organizational-accountability finding, documented elsewhere in this corpus, that even a fully auditable, deterministic automated decision still creates a human 'explain-this-to-the-customer' burden, and that technical explainability alone is insufficient without the surrounding organizational and relational structures.
Read next: extract-when-employees-are-held-accountable-for-ai-generated-decisions — shows employees masking or reinterpreting even well-audited automated decisions when they lack the organizational structures to explain them credibly to the people affected.
Closes gaps elsewhere
- Podcast: Will Agentic AI Bring Fantasia's Sorcerer's Apprentice to Life?: A Conversation with Tracy Bannon · g5 — Offers a concrete engineering answer to the named gap (no engagement with specification-based conformance testing for non-deterministic agents): DMN is itself an open specification, and the demo shows generating use/test cases directly from a published model ('Can you create use cases for this model?') for conformance checking.
- Research: Why You Shouldn't Treat AI Agents Like Employees · g5 — Demonstrates a concrete formal audit mechanism — a published, versioned DMN decision model with a governed review/approve cycle and a full execution trace from input to output — as exactly the kind of accountability infrastructure that gap flags as missing from the 'AI cannot be held accountable' framing.
Related documents
- AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering
- When Employees Are Held Accountable for AI-Generated Decisions
- To Adopt AI at Scale, Employees Need to Trust Agents
- Your AI agent just blamed the network team. Now what?
- Presentation: Platform Engineering in the Age of AI
- Software Teaming in the Age of AI with James Herr and Woody Zuill
- The Economics of Agentic AI: Engineering for Imperfection
- Agent Skills: Google-Backed AI Coding Agent Workflow
- Article: Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules
- Beyond the 200 OK: Architecting Observability for AI