LLM-as-Judge
Using an LLM to grade another LLM's or agent's output against a rubric or human-labeled examples — a specific evals technique for subjective/non-deterministic quality dimensions, requiring judge-model selection, calibration against human labels (e.g. TPR/TNR), and drift monitoring to stay trustworthy.
Also known as: LLM-as-judge, LLM judge, AI judge, automated judge, judge model, judge (evaluator)
Part of: Evals
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| A pragmatic guide to LLM evals for devs The Pragmatic Engineer · 2025-12-02 | LLM-as-judge | An AI evaluator built to automate a domain expert's subjective judgment call (e.g., 'should the agent hand off to a human here?') at scale, trained/validated against a hand-labeled golden dataset of PASS/FAIL decisions plus detailed human critiques — explicitly NOT a generic off-the-shelf 'helpfulness'/'factuality' scorer. |
Documents (1)
- A pragmatic guide to LLM evals for devs The Pragmatic Engineer · 2025-12-02 Reader 🖥️
LLM evaluation cannot be bolted on as generic scored metrics or traditional deterministic unit tests; it must be built bottom-up from systematic error analysis of real interaction traces, using code-based assertions for objectively-checkable failures and a human-calibrated LLM-as-judge for subjective ones.
Propositions touching this concept (11)
By publication month, oldest first (extraction month when no publication date is known).
2025-12
- explicitThe correct evaluator type depends on whether a failure is objective/deterministic (use a code-based assertion) or subjective/judgment-based (use an LLM-as-judge trained on human-labeled examples) — using the wrong tool for either case wastes effort or produces unreliable results.
- implicitA trustworthy LLM-as-judge requires validating the judge's agreement with human expert judgment (implied via TPR/TNR-style metrics), rather than simply deploying an LLM to grade outputs and trusting its scores.
2026-06
- explicitestablishedThe real engineering-leadership skill now is evaluating which model fits a specific use case, quality bar, and price point; industry benchmarks are useful only for orientation, so companies need their own task-specific evals, ideally a manual judge paired with an LLM-as-judge to scale.
- explicitNo single evaluation method covers all seven dimensions; production pipelines must combine automated functional testing, security/safety scanning, LLM-as-judge/agent-as-judge, browser-based testing, trajectory inspection, human review, online sampling, and standardised benchmarks, each mapped to the dimensions it is actually suited for.
2026-07
- explicitcorroboratedVerification is what separates toy demos from production agents: rules-based feedback (tests/linters/type checkers), visual feedback (screenshots), and LLM-as-judge (a separate subagent); giving the model a way to verify its own work improves quality 2-3x (Cherny), and verification is best framed as guides (feedforward) versus sensors (feedback).
- explicitEvals should be stored in the registry as an 'eval pack' — a typed dataset (input schema, output schema), an eval runner, and an LLM-as-judge scoring config — functioning as an inductive, example-based product-requirements-document that non-data-scientist business people can author and read.
- explicitAn LLM-as-judge should use a different, and can use a much smaller/cheaper, model than the agent it is judging -- both because a same-model judge tends to agree with itself, and because judging output against a rubric is a simpler task than generating it.
2026-08
- explicitThe judge model evaluating a change should be isolated from the agent that produced the code being judged, to control for self-enhancement bias (a model favoring patterns similar to what it generates).
2026-09
- explicitdevelopedGrounding an LLM-as-judge evaluation in a specific simulation scenario with a known, definable goal allows task completion to be verified automatically by rule-based checks against real backend state, which is more trustworthy than asking an LLM to judge an open-ended conversation without a defined ground truth.
- explicitParallelizing agent work should come last, because parallelism multiplies whatever review bottleneck already exists; a single unit of work needs a dependable automated judge, a cheap recovery path, and an absorbable review format before it is replicated many times over.
2026-10
- explicitOwning your control flow — the prompt, the switch-statement over model output, the context-window builder, and the loop-exit logic — is what gives an engineer levers (break, switch, summarize, LLM-as-judge) that a framework's hidden internal loop does not.