12-Factor Agents: Patterns of reliable LLM applications — Dex Horthy, HumanLayer

AI Engineer · video · 2026-10-08 Reader 🖥️

practical · argumentative — A conference talk distilling a published '12-factor agents' GitHub framework (borrowing the 12-Factor App naming convention) into a set of prescriptive engineering patterns for building production LLM agents, argued from the speaker's experience interviewing 100+ founders/builders.

Structure

Key terms

• 12-factor agents
  A set of patterns (deliberately modeled on Heroku's 12-Factor App) for building LLM-based applications that are reliable enough for production, extracted inductively from observing what successful builders were actually doing to existing codebases rather than prescribed top-down.
• tool use is harmful
  A deliberately provocative framing (parodying Dijkstra's 'Goto Considered Harmful') arguing that treating tool calls as a magical, ethereal agent-environment interaction obscures the fact that a tool call is just an LLM emitting JSON that deterministic code then acts on — demystifying it is what lets you engineer it well.
• owning your control flow
  Keeping the loop/switch-statement/context-window-building logic as code you write and can modify, rather than ceding it to a framework's internal agent loop — because only this gives you levers like break, switch, summarize, or LLM-as-judge inside the loop.
• micro agents
  Small agent loops of roughly three to ten steps embedded inside an otherwise mostly deterministic DAG/pipeline, handling one bounded decision point rather than the whole workflow.
• LLMs are pure/stateless functions
  The model itself carries no memory between calls and cannot be made 'smarter' mid-task; the only lever an engineer has over output quality is what tokens are placed into the context window on each call.

Propositions

• explicitThe most successful production LLM agents observed across 100+ founder/builder conversations were mostly ordinary software with a handful of specific LLM-call patterns applied to existing codebases, not agentic greenfield rewrites.conf: high
• explicitThe single genuinely magical capability of an LLM, from which all other agent functionality is built, is turning natural language into structured JSON.conf: high
• explicitA tool call is not a qualitatively different, ethereal agent-environment interaction; it is an LLM emitting JSON that deterministic code then acts on, structurally identical to any other JSON-producing LLM call.conf: high
• explicitLetting an LLM freely determine its own next step in an unbounded loop (the naive 'just tell it the goal' agent model) breaks down in practice, principally because of long-context-window degradation as workflows lengthen.conf: high
• explicitOwning your control flow — the prompt, the switch-statement over model output, the context-window builder, and the loop-exit logic — is what gives an engineer levers (break, switch, summarize, LLM-as-judge) that a framework's hidden internal loop does not.conf: high
  supportsNo Vibes Allowed: Solving Hard Problems in Complex Codebases – Dex Horthy, HumanLayer · p6 — This talk's 'own your control flow' / sub-agents-only-for-isolation stance anticipates the later essay's explicit rejection of anthropomorphized sub-agent roles in favor of context-isolation-only sub-agent use.
• explicitAn agent's execution state (step, retries) and business state (messages, pending approvals) should be modeled explicitly and persisted via a state ID so the agent loop can be launched, paused, and resumed like any standard API — including pausing for long-running tool calls.conf: high
  supportsNo Vibes Allowed: Solving Hard Problems in Complex Codebases – Dex Horthy, HumanLayer · p3 — Same author's later essay restates and deepens the 'LLMs are stateless, only the tokens you control can be engineered' premise with explicit context-rot data (the 'smart zone'/'dumb zone' finding) that this talk only asserts.
• explicitBeyond a certain quality bar, prompts must be hand-written token-by-token rather than generated via a library's default primitives, because reliability is entirely a function of token quality and only careful manual control over input tokens can push past that bar.conf: high
• explicitContext-window construction should not be limited to the standard chat-message format; information can be stringified and modeled however the builder chooses, and curating/compressing error content (rather than dumping full stack traces) measurably improves recovery from failed tool calls.conf: medium
• explicitPushing the tool-call-vs-message-to-human decision onto the model's very first generated token (rather than treating human contact as a separate downstream branch) is an underused, subtle technique that most builders avoid despite its value.conf: medium
• explicitThe working shape for reliable production agents is 'micro agents' — small loops of roughly three to ten steps for one bounded decision — embedded inside a mostly deterministic surrounding pipeline, rather than one large agent owning an entire workflow.conf: high
  supportsAdvanced Context Engineering for Coding Agents (ace-fca.md) · p3 — This talk's 'micro agents inside a mostly deterministic DAG' is the architectural precursor to the later essay's formalized 40-60% context-utilization, frequent-intentional-compaction workflow — same author, earlier and coarser-grained version of the same practice.
• implicitAs models get more capable, the deterministic-code share of a pipeline will shrink and the agent-owned share will grow over time, but the engineering discipline of controlling context quality remains necessary regardless of model capability.conf: medium
• explicitFrameworks that abstract away the 'hard AI parts' (prompting, control flow, token curation) are solving the wrong problem; frameworks should instead absorb the hard non-AI parts (infrastructure, state persistence, protocol plumbing) so builders can spend their effort on the AI-hard parts.conf: high

Arguments

Narratives

• The speaker's own early DevOps agent: given a Makefile and told to 'build the project,' it executed steps in the wrong order; two hours of iterative prompt-fixing eventually produced a reliable prompt — at which point the speaker realized a 90-second bash script would have solved the underlying problem.
  → Supports the proposition that not every problem needs an agent, and frames the entire talk's motivating tension: effortful agent engineering sometimes reveals the task didn't need an agent at all.
• The HumanLayer internal deploy bot: a mostly-deterministic CI/CD pipeline hands a single ordering decision (deploy frontend or backend first) to a small agent loop; a human intervenes mid-loop to override the order; the pipeline resumes deterministically to run end-to-end tests, falling back to a rollback agent on failure.
  → Concretely illustrates 'micro agents embedded in a deterministic DAG,' pause/resume via state, and human-in-the-loop override — the talk's central architectural recommendation made tangible.

Questions addressed

• ✅ Why do agents that reach 70-80% quality on a task stall there instead of improving further with better prompting alone?
  Because past that threshold, further gains require hand-engineering individual tokens/context rather than iterating on a framework-generated prompt — a qualitatively different (not just more effortful) activity.
• ✅ What should a framework for building LLM agents actually provide, given that frameworks are widely criticized?
  Infrastructure/plumbing (state persistence, pause/resume, protocol handling) rather than the agent's reasoning loop itself — the talk frames this as a 'wish list,' not a rejection of frameworks as such.
• ⭕ How should an agent decide between acting (calling a tool) and escalating to a human?
  A technique (push the decision to the model's first generated token) is offered as a means to make this choice more legible to the model, but the talk does not resolve when/how often this decision should favor the human versus the tool.

Gaps

• incompleteThe talk does not address how a team coordinates or reviews this hand-engineered, token-by-token prompt and control-flow discipline across multiple engineers over time — it is framed entirely from a single practitioner's vantage point, with no discussion of code review, documentation, or onboarding for the 'owned' control flow and prompts.
  Read next: extract-humanlayer-advanced-context-engineering-ace-fca.json and extract-horthy-no-vibes-allowed-complex-codebases.json (same author's later, team-scale essays) take up exactly this gap — team mental alignment, review at the plan/research layer, and scaling the practice beyond a single motivated practitioner.
• incompleteThe claim that 'tool use is harmful' is framed rhetorically but the talk does not examine any downside of the demystified view itself — e.g., whether treating tool calls as 'just JSON and code' obscures real risks (prompt injection via tool outputs, tool-call hallucination against live systems) that a 'the model is doing something special' framing might have made more salient as a safety concern.
  Read next: Compare against the corpus's agent-security and lethal-trifecta material, which treats tool-call outputs as an attack surface rather than purely an engineering-reliability concern.

Related documents

• AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering
  Agent Harness Agentic Loop Agentic vs. Deterministic System Design Context Engineering
• The Anatomy of an Agent Harness
  Agent Harness Agent Tool Management Agentic Loop Context Engineering
• What is “loop engineering?”
  Agent Harness Agentic Loop Context Engineering Human-in-the-Loop
• 11 Tips For AI Coding With Ralph Wiggum
  Agent Harness Agentic Loop Human-in-the-Loop
• Software Factories, Light and Dark
  Agent Harness Agentic Loop Human-in-the-Loop
• Agent Harness Engineering
  Agent Harness Agent Tool Management Agentic Loop
• Don't train the model, evolve the harness
  Agent Harness Agentic Loop Context Engineering
• Designing a Multi-Agent System for Engineering Support at Scale: A Case Study From Grab
  Agent Tool Management Context Engineering Human-in-the-Loop
• No Vibes Allowed: Solving Hard Problems in Complex Codebases – Dex Horthy, HumanLayer
  Agent Harness Context Engineering Human-in-the-Loop
• Loop Engineering: Building Blocks, Adoption, and Impact
  Agent Harness Agentic Loop Human-in-the-Loop