Presentation: From Thousands to One: Building LLM-Powered Selection Systems
Jendrik Jördening · rss · 2026-08-17 source ↗ Reader 🖥️
Agent ObservabilityAgentic vs. Deterministic System DesignGuardrailsLLM Reliability FailuresNon-Deterministic Software
When you need an LLM to select one item from among many database-backed options (an ID, a station, a record), you must strip the model down to only the irreducibly textual/semantic step it is actually good at — schema-restricted, range-bounded, and independently validated by a second model call — while pushing every deterministic operation (filtering, comparison, control flow) into ordinary code, recovering a classic model-view-controller separation that a single freeform LLM call destroys.
practical · mixed — A practical, rules-based engineering talk (six numbered rules plus a running conclusion telling the audience what to do) delivered through a single extended narrative case study — a live-rebuilt pipeline for finding a delayed train's actual arrival time from a screenshot — rather than through abstract exposition.
Structure
| Part | Role |
|---|---|
| Opening: origin story from procurement automation (choosing a cost-center ID) and the pre-LLM argmax-classifier baseline | context |
| Diagnosis of why LLMs break the classic model/view/controller separation for decision systems (conversation history as 'model', no == comparison, context-window ceiling, prompt injection, hidden MCP calls, random information retrieval) | problem statement |
| Extended live case study: using a screenshot of a delayed Deutsche Bahn train, an EVA-station-number lookup API, and a timetable API to automate a delay-reimbursement claim | evidence / running example |
| Rule 0 (Remove Randomness) through Rule 6 (Narrow Down Deterministically), each demonstrated as a fix to a failure just shown in the live demo | evidence / prescriptive framework |
| Conclusion: naming the tradeoff (rigidity/state-machine cost vs. reliability/metrics payoff) and reframing the result as MVC restored | conclusion |
| Q&A: audience presses on which steps are actually deterministic vs. LLM-driven, and whether validator feedback loops back into re-prediction | clarification / stress-test of the framework |
Key terms
- Rule 0: Remove Randomness
The mandatory first step before any other reliability work: set temperature to 0, fix random seeds, and disable 'thinking budget' unless the task actually needs it — while acknowledging that some residual randomness survives because the network's nonlinearity means even 'deterministic' settings don't guarantee identical output on rerun. - Discriminator
A second, independent LLM call whose only job is to check whether a specific structured choice the first model made is correct (true/false) — e.g., 'is this the station the user meant?' — giving a live accuracy signal separate from the original prediction. - Model-View-Controller (MVC), reapplied
The pipeline's persistent state (e.g., the extracted screenshot data, the EVA number found so far) is the model; the chat-style interface business sees is the view; a controller looks at current state and target state and asks the LLM to perform only the one narrow transition needed to move between them. - Narrowing down (deterministically)
Filtering the candidate set with ordinary code before the LLM ever sees it — e.g., 'all trains departing after time X to destination Y' instead of asking the model to consider every train in Germany for a year — so the LLM is choosing among a small, well-posed set rather than an unbounded one. - Map during predict
The fallback for when the raw inputs needed for a mapping decision aren't cleanly separable in advance (e.g., classifying whether an image shows a delayed-train screenshot at all) — in that case, do the mapping and the extraction in the same LLM call rather than splitting them into two.
Propositions
- explicitLLM-driven decision pipelines collapse the classic model/view/controller separation into one blob, because the effective 'model' becomes the raw conversation history — which is a poor fit for a database that needs an exact, typed ID.conf: high
- explicitFreeform LLM text output cannot be compared to ground truth with a simple equality check, which makes integrating LLM decisions with existing accuracy/observability infrastructure difficult.conf: high
- explicitThe context window imposes a hard ceiling on how many candidate options an LLM can be asked to choose among in a single call; a large historic option set makes the context 'explode'.conf: high
- explicitBecause these pipelines ingest untrusted natural-language input (e.g., invoice text), they introduce a new attack surface — prompt injection — that existing alerting/monitoring tooling isn't built to trace, especially through hidden MCP tool calls.conf: high
- explicitIf a JSON schema field is not explicitly marked nullable, the model will fabricate a plausible non-null value rather than correctly indicating the information is absent.conf: high
- explicitConstraining structured output to an enum with too many members (empirically around 500+) causes the model provider's API to reject the request outright, independent of the model's own reasoning ability.conf: high
- explicitRestricting a model's output to a bounded integer ID range (0 to N) rather than an enum of human-readable names is a more robust way to force a valid, database-comparable choice, and scales past the branching limits that break large enums.conf: high
- explicitA second 'discriminator' LLM call that independently validates a structured choice (true/false) produces a live accuracy signal you can track as a metric without needing labeled ground truth for every prediction, though early on you should still spot-check a handful of cases manually.conf: high
- explicitcorroboratedThe right division of labor is to give the LLM only tasks that are irreducibly textual/semantic (fuzzy-matching misspelled names, reading values off an image, judging which of several paths a train actually took) and to do everything numeric, comparative, or deterministic — like finding the closest departure time — in ordinary code.conf: highextendsAI Root Cause Analysis Shifts from Model Reasoning to Context Engineering · p3 — Extends the RCA article's claim that deterministic pipelines enable unambiguous model-vs-harness attribution: here, separating the LLM's textual step from deterministic code plus a discriminator call similarly yields clean, database-comparable per-decision accuracy signals in a structured-selection (not incident-diagnosis) domain.supports ←Presentation: Decision Models in Agentic Architectures: From Production to Agent Skills · p9 — Both converge on the same division of labor: give the LLM only the irreducibly textual/ambiguous part and force the actual classification/decision through deterministic, non-LLM logic.
- implicitNarrowing the candidate set with deterministic code before the LLM ever sees it is itself the reliability intervention, not merely a performance optimization — the same underlying LLM call that fails against an unbounded option set succeeds once the set is pre-filtered to a small, well-posed one.conf: highsupportsAI Root Cause Analysis Shifts from Model Reasoning to Context Engineering · p7 — Independently arrives, in a different domain (mapping free text to database IDs vs. SRE root-cause analysis), at the same architectural conclusion the RCA article reports as an industry trend: replace end-to-end agentic decision-making with deterministic narrowing plus a narrow, bounded LLM step.
- explicitThis architecture trades away the flexibility of prompt-only iteration for the rigidity of a fixed state machine: any new business requirement now requires a code change (a new state/transition) rather than a prompt edit — a price the author judges worth paying.conf: high
- implicitBusiness stakeholders accept an architecture regardless of how deterministic it is underneath, as long as the user-facing view still resembles a chat interface — the demand for 'AI' is really a demand for a specific interaction surface, not for autonomous reasoning underneath.conf: medium
Arguments
Schema design (nullable fields, bounded numeric ranges instead of free-text or oversized enums) should be treated as a first-class reliability lever in production LLM systems, on par with prompt wording. strength: medium
Explicit premises:
- A schema without a nullable field causes the model to fabricate values instead of correctly returning 'none'.
- An enum with too many members fails at the API level regardless of model capability.
- An ID bounded to a numeric range guarantees the output is at least mechanically valid and database-comparable.
Implicit premises:
- The model is more reliable at obeying formal, mechanically-checkable constraints (type, range) than semantic instructions phrased in the prompt (e.g., 'only say none if truly missing').
- Mechanical validity of the schema constraint does not by itself guarantee the underlying decision is correct — schema restriction bounds the failure mode, it doesn't eliminate it.
Inferences:
- Teams evaluating LLM-selection reliability should audit schema design (nullability, enum size, range bounds) before tuning prompts or switching models.
Pairing deterministic pre-narrowing with a discriminator validation call gives a production system usable accuracy metrics without requiring hand-labeled ground truth for every decision. strength: medium
Explicit premises:
- The discriminator returns true/false against the specific decision made.
- This produces two independently trackable signals: the predictor's output and the validator's agreement/disagreement rate.
- Manual spot-checks in the early phase catch systemic problems before full automation.
Implicit premises:
- Disagreement between predictor and validator is informative even without ground truth — but this assumes the two calls fail somewhat independently.
- The validator, itself an LLM (potentially the same underlying model or family), is treated as a meaningfully independent check rather than one that could share the predictor's blind spots.
Inferences:
- Predictor/validator agreement rate could be reported to stakeholders as a proxy accuracy metric even before ground truth exists — but this inference depends on an error-independence assumption the talk never tests.
Narratives
- The entire talk is structured as one running case study: the author, delayed on a Deutsche Bahn train from Bregenz to Munich, builds a live pipeline (screenshot → EVA station number → timetable API → next-departure lookup) to automate his own delay-reimbursement claim, hitting and then fixing a sequence of concrete failures (looping through 'Ulm, Ulm, Ulm', hallucinated routes through Dortmund and a fictional map through the Baltic Sea, a Deutsche Bahn API error message from an oversized enum schema).
→ Makes each abstract rule (0 through 6) concrete by showing the specific failure it fixes, rather than arguing for the rules in the abstract — the narrative itself is the argument for why each rule is necessary. - A two-year-old, barely-viewed LinkedIn post about hitting the same 'too much branching' enum-size error against Azure OpenAI endpoints, told as an aside when the same error recurs live in the demo.
→ Implicitly supports the proposition that structured-output branching limits are a known, longstanding, cross-provider constraint rather than a new or provider-specific quirk — lending credibility to Rule 2 (restrict your schemas) beyond the single live demo.
Questions addressed
- ✅ How do you get an LLM to reliably select one item (e.g., a database ID) from a large set of options without hallucination or drift?
Schema restriction (nullable fields, bounded ID ranges instead of oversized enums) plus deterministic pre-narrowing of the candidate set plus a discriminator validation call. - ✅ How do you get accuracy/observability metrics out of an LLM-driven decision system when you can't do an == string comparison?
Force the output into a comparable typed ID, and add a discriminator call that produces an independent true/false signal you can track as a metric. - ✅ Should an LLM ever be trusted to perform numeric/deterministic comparisons directly (e.g., picking the soonest of several departures)?
No — explicitly answered in the Q&A: narrow the candidate set and do the final comparison deterministically in code; only give the LLM the parts that are genuinely textual/semantic. - ⭕ How do you know the discriminator's true/false validation judgments are themselves reliable, especially if predictor and validator share the same underlying model or training data and could fail in correlated ways?
- ⭕ What is the real-world cost, in engineering time and change-management overhead, of the rigidity this architecture introduces — i.e., how often and how expensively does 'you actually need to touch code again' actually bite in practice?
Gaps
- incompleteNo discussion of how to validate the discriminator/validator itself. If the predictor and validator calls use the same base model (or correlated training data), their errors may not be independent, which would undermine the claim that predictor/validator disagreement is a meaningful, ground-truth-free accuracy signal.
Read next: Look for LLM-as-judge calibration and independence studies — work examining whether a second LLM call genuinely catches errors the first one made, versus reproducing the same blind spots. - incompleteEvery 'it works now' claim in the talk is demonstrated by a single live rerun of the pipeline rather than a systematic evaluation across many screenshots/transcripts, so the reliability improvement attributed to each rule is asserted rather than measured (and the talk itself notes that even the 'deterministic' rerun changed output between runs).
Read next: Look for a systematic before/after accuracy evaluation of schema-restriction and narrowing techniques on a labeled dataset, rather than an anecdotal demo. - incompleteNo treatment of the cost or latency overhead of the resulting multi-call pipeline (predict, map, validate, narrow, controller step) compared to a single end-to-end LLM call — the rigidity/reliability tradeoff is discussed explicitly, but the compute-cost tradeoff of running several models per decision is not.
Read next: Look for production case studies that report per-decision latency and token cost for multi-step 'predict + validate' pipelines versus single-call approaches. - incompleteRule 6 ('narrow down deterministically') assumes a pre-existing, well-structured backing dataset to narrow into (a station table with EVA numbers). The talk doesn't address what to do when the candidate set itself is fuzzy, unbounded, or must be discovered dynamically — which is exactly the harder procurement-cost-center use case the author mentions in the opening and then explicitly declines to return to.
Read next: Look for techniques for LLM-driven selection when the candidate universe cannot be cheaply pre-filtered by deterministic code (e.g., open-world entity resolution, dynamic vendor/cost-center taxonomies).
Related documents
- AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering
- Beyond the 200 OK: Architecting Observability for AI
- The Economics of Agentic AI: Engineering for Imperfection
- Your AI agent just blamed the network team. Now what?
- This coding agent doesn’t want your feedback — it ships without it
- Gene Kim on Vibe Coding - Part III | Episode 412 | Troubleshooting Agile
- Presentation: Platform Engineering in the Age of AI
- Looking back at the Future of Software Engineering unconference
- Presentation: Decision Models in Agentic Architectures: From Production to Agent Skills
- Session Traces and Cost Controls Help Diagnose AI Agent Failures