A pragmatic guide to LLM evals for devs

The Pragmatic Engineer · email · 2025-12-02 Reader 🖥️

LLM evaluation cannot be bolted on as generic scored metrics or traditional deterministic unit tests; it must be built bottom-up from systematic error analysis of real interaction traces, using code-based assertions for objectively-checkable failures and a human-calibrated LLM-as-judge for subjective ones.

practical · expository — A how-to guide (guest-written by Hamel Husain via Gergely Orosz's newsletter) walking developers through a concrete workflow for building LLM evals — argues for a specific practice, illustrated with a single running case study (NurtureBoss).

Structure

Part	Role
The 'vibe-check' trap and the Three Gulfs framing (section 1)	Problem statement
Error analysis workflow: data viewer, open coding, axial coding, prioritization by frequency (section 2)	Core method / evidence
Code-based evals vs. LLM-as-judge, matched to deterministic vs. subjective failure types (section 3)	Method — tool selection
Building an LLM-as-judge from a hand-labeled, PASS/FAIL golden dataset (section 4, truncated)	Method continuation, cut off mid-explanation by paywall
Judge validation via TPR/TNR against human labels; CI/CD + production monitoring; the 'flywheel of improvement' (sections 5-7, only outline-level visible)	Promised conclusion / operationalization — not available in this capture

Key terms

Vibes-based development
The failure mode of iterating an LLM prompt by eyeballing a few outputs and shipping when it 'looks good to me' (LGTM), with no systematic way to know if a change actually helped or broke something.
Three Gulfs (Comprehension, Specification, Generalization)
A framework for why LLM development is hard: the Gulf of Comprehension (developer can't manually inspect all data/behavior at scale), the Gulf of Specification (prompts underspecify true intent, forcing the model to guess), and the Gulf of Generalization (even a well-specified prompt won't reliably generalize to all inputs).
Error analysis
A bottom-up, qualitative-then-quantitative discipline — reviewing real interaction traces, writing open-ended notes on failures (open coding), clustering them into themes (axial coding), then counting frequency to prioritize — borrowed from decades of ML practice and grounded theory in social science.
Open coding / axial coding
Open coding: writing free-form, non-checklist-constrained notes on any observed undesirable behavior in a trace. Axial coding: grouping those open-ended notes into a smaller number (5-10) of thematic buckets, with an LLM assisting the initial clustering but a human reviewing and finalizing categories.
Code-based evals
Assertion-style tests against a 'golden dataset' of input/expected-output pairs, used only for failures that have one objectively correct answer (e.g., parsing a date). Cheaper to build/maintain and can run on every commit.
LLM-as-judge LLM-as-Judge
An AI evaluator built to automate a domain expert's subjective judgment call (e.g., 'should the agent hand off to a human here?') at scale, trained/validated against a hand-labeled golden dataset of PASS/FAIL decisions plus detailed human critiques — explicitly NOT a generic off-the-shelf 'helpfulness'/'factuality' scorer.
Golden dataset Evals
A curated set of test cases — for code-based evals, input plus one definitively correct expected output; for LLM-as-judge, input/trace plus a human expert's PASS/FAIL verdict and a written critique explaining the reasoning.
Eval Engineering
NOT FOUND. The exact phrase 'eval engineering' (or 'evals engineering') does not appear anywhere in the visible text. The article uses 'evals' as the general noun throughout, refers to Hamel Husain's course as 'AI Evals For Engineers & PMs' and his forthcoming book as 'Evals for AI Engineers' — but never names 'Eval Engineering' as a distinct discipline, role, or practice separate from the general concept of evals. The closest framing is the article's opening line calling evals 'a new toolset for any and all AI engineers,' i.e. evals-as-a-skill-within-engineering, not a named sub-discipline.

Propositions

explicitTest-Driven Development, applied naively to LLM outputs, fails because TDD assumes a single deterministic correct output to assert against, while LLM outputs are subjective and have no single correct answer.conf: high
supportsYou can't unit test for taste · p1 — Both independently argue that red/green, single-correct-answer testing breaks down once there's no ground truth or ambiguous/subjective output to check against.
explicitGeneric, pre-built evaluation metrics (e.g. 'helpfulness', 'factuality' scored 1-5) are often worse than useless because they don't correlate with what actually matters to users and create a false sense of security.conf: high
supportsPodcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook · p2 — Both make the same claim from different angles: the highest-value evals are built from an organization's own data/domain expertise, not generic off-the-shelf benchmarks/metrics.
explicitSystematic review of real interaction traces (error analysis) — not predefined error checklists — is the highest-ROI activity in AI application development, because it surfaces the actual failure modes rather than assumed ones.conf: high
explicitBinary PASS/FAIL judgments produce more reliable, actionable evaluation data than Likert-scale (1-5) ratings, because point ratings are subjective and inconsistent across reviewers while binary decisions force a clear line and a clear signal to fix a bug.conf: high
supportsPodcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook · p3 — Both position evals as a distinct, higher-stakes discipline than traditional software testing for AI systems — this article supplies the concrete binary-PASS/FAIL mechanism for why that discipline looks different.
explicitThe correct evaluator type depends on whether a failure is objective/deterministic (use a code-based assertion) or subjective/judgment-based (use an LLM-as-judge trained on human-labeled examples) — using the wrong tool for either case wastes effort or produces unreliable results.conf: high
extendsPresentation: From AI Agent Demo to Production: Automated Testing and Evaluation · p8 — The other document grounds LLM-as-judge validity in a specific simulation scenario with a known goal; this article extends that with the concrete workflow for producing the human-labeled golden dataset (PASS/FAIL + critique) a judge should be built and calibrated from.
explicitAn investment of a few hours building a custom, domain-tailored data viewer materially increases the speed and quality of manual trace review compared to generic off-the-shelf observability tools.conf: high
implicitA trustworthy LLM-as-judge requires validating the judge's agreement with human expert judgment (implied via TPR/TNR-style metrics), rather than simply deploying an LLM to grade outputs and trusting its scores.conf: medium

Arguments

Software engineers building or shipping LLM-based features should adopt a structured error-analysis-first evals workflow rather than either ad hoc vibe-checking or naive unit-test-style TDD. strength: high

Explicit premises:

LLM outputs are non-deterministic and often have no single correct answer.
Vibes-based iteration ('LGTM, ship it') doesn't scale and doesn't catch regressions.
Generic off-the-shelf metrics don't correlate with real user-facing quality.
Bottom-up error analysis (reviewing real traces, open/axial coding) surfaces the failure modes that actually matter, ranked by frequency.

Implicit premises:

Developers have (or can get) access to enough real or synthetic interaction traces to analyze.
A domain expert exists and is willing to spend the time hand-labeling and critiquing example failures.
The cost of building a custom trace viewer and running manual error analysis is justified by the downstream savings from catching the right failure modes early.

Inferences:

Teams without a domain expert willing to do hands-on labeling, or without enough real traffic to generate traces, may not be able to execute this workflow as described.
The 'evals' skill set described here is effectively data-science-adjacent (qualitative coding, dataset curation, statistical validation) layered onto software engineering, not a pure engineering skill.

Narratives

NurtureBoss, an AI leasing-assistant startup, moves from shipping-on-vibes to a systematic error-analysis workflow: they build a custom data viewer, open-code and axial-code hundreds of traces, discover that three issues (date handling, handoff failures, conversation flow) account for most problems, then build a code-based eval for date parsing and a hand-labeled golden dataset (with PASS/FAIL + critiques) for the more subjective handoff-timing problem.
→ Carries the entire argument concretely — every abstract claim in the article (vibes-trap, error analysis, code-based vs. LLM-judge split) is demonstrated through this single running example rather than argued in the abstract.
A mental-health startup's evaluation dashboard filled with generic 'helpfulness'/'factuality' 1-5 scores looked impressive but was unactionable — reviewers couldn't explain what separated a '3' from a '4', and the scores didn't track user satisfaction.
→ Supports the proposition that generic pre-built metrics are often worse than useless, and motivates the binary-PASS/FAIL design choice.

Questions addressed

✅ How should developers evaluate LLM-based application quality when outputs are non-deterministic and there's no single 'correct' answer to assert against?
Via error analysis to discover real failure modes from data, then routing each failure type to either a code-based eval (deterministic) or a calibrated LLM-as-judge (subjective), both built from hand-curated golden datasets.
✅ What if there isn't enough real user data yet to run error analysis on?
Use a powerful LLM to generate synthetic, realistic user queries covering target scenarios and edge cases, bootstrapping error analysis before real users exist — though the article explicitly defers the how-to detail to its paid course.
⭕ How do you know the LLM-as-judge itself can be trusted — i.e., that it agrees with human judgment and doesn't drift or share blind spots with the system it's grading?
Only gestured at via the outline ('validate against human expertise... TPR and TNR') — the elaborated method sits in the paywalled section 5, unavailable in this capture.
⭕ How do evals fit operationally into CI/CD and post-deployment production monitoring, and how does the whole cycle repeat (the 'flywheel of improvement')?
Not resolved in the visible text — only promised in the outline (sections 6-7), which state evals should run in CI/CD but be continuously validated against production data, cycling through analyze → measure → improve → automate → repeat.

Gaps

incompleteThe article's own central risk — that an LLM-as-judge might be unreliable, drift over time, or share blind spots with the agent it grades — is named in the outline but its resolution (the TPR/TNR-based human-calibration method) is entirely paywalled in this capture, so the reader is left with the problem statement and the destination without the actual mechanics. This is the same open risk already flagged in extract-ai-agent-demo-to-production-testing-evaluation#g2 (LLM-as-judge reliability asserted, not demonstrated) — this article names a plausible metric (TPR/TNR against human labels) but does not demonstrate it, so it sharpens rather than closes that gap.
Read next: Read the unlocked full article at https://newsletter.pragmaticengineer.com/p/evals (subscriber access required), or Hamel Husain's own public writing on LLM-as-judge alignment/calibration.
incompleteNo discussion of the cost/effort trade-off of this workflow at the team level: how much analyst/domain-expert time error analysis and golden-dataset curation actually consumes relative to the engineering time it saves, or how this scales when no single domain expert like 'Jacob' exists or is willing to hand-label hundreds of traces.
Read next: Look for practitioner writeups quantifying the labor cost of error-analysis-driven eval-building versus generic eval tooling.
uninformedThe article asserts (via the Three Gulfs framing) that naive TDD 'falls short' for LLMs without engaging the corpus's own counter-evidence that code-based evals are themselves just traditional assertion-style unit tests re-applied to the subset of LLM behavior that IS deterministic — i.e., TDD isn't abandoned so much as partitioned, a point the article makes implicitly but never states as a correction to its own TDD framing.
Read next: extract-you-cant-unit-test-for-taste (this corpus) — direct treatment of where unit-testing does and doesn't apply to AI-generated output quality.