The Almedra Token-Efficiency Benchmark, Updated For GLM 5.2

Lucena.One · article · 2026-07-13 Reader 🖥️

Agent Harness AI Unit Economics Coding Agents Context Engineering Evals Token Efficiency Vendor Research Conflict of Interest

Running the same prompt and repo through six different coding-agent harnesses against the identical GLM 5.2 model, Lucena Coder again produced a working result using roughly a third to a fifteenth of the tokens and cost of its rivals, showing that harness design — not the underlying model — is the dominant lever on coding-agent token efficiency, and that upgrading the model (GLM 5.1 to 5.2) narrows but does not close that gap.

practical · expository article with an empirical/benchmark backbone — Reads as a vendor blog post (Lucena.One, maker of 'Lucena Coder') built around a small, self-run head-to-head benchmark. It presents quantitative usage/cost data (via OpenRouter logs) as evidence for a practical recommendation about which coding-agent harness to use, rather than developing a theory.

Structure

Part	Role
Benchmark setup (Almedra fixture, identical prompt/repo/model)	establishes a controlled comparison
The Result table (cost, tokens, calls, time, pass/fail per agent)	presents the comparative headline data
The Prompt (verbatim task spec)	shows the exact task given to every agent
What We Verified (correctness checklist)	establishes a correctness gate before cost is compared
The Ledger (input/output/cached/fresh token breakdown, OpenRouter logs)	decomposes where each agent's cost comes from
What This Test Tells Us (interpretation)	draws the headline conclusion
Time to Completion (caveat on measurement precision)	flags a methodological limitation in one metric
Run The Harness (link to benchmark repo and prompt)	invites third-party replication

Key terms

token efficiency (in this benchmark) Token Efficiency
Total billed OpenRouter cost and total token count (input + output, cached + fresh, across all model calls) needed to reach a verified, buildable result for one fixed feature request.
Almedra
The name of the fixed benchmark fixture: 'Almedra Timepieces,' a one-page React/Vite small-business website repo used as the constant test bed across benchmark runs and model versions.
cached vs. fresh tokens
The Ledger table's split of each agent's input tokens into cache reads (cheaper, reused context) and fresh tokens (newly processed context) — used to argue that the size of what's sent, not just cache-hit rate, drives cost.
the field compressed
The author's characterization of the GLM 5.1-to-5.2 update narrowing (but not eliminating) the cost/token spread across the six harnesses compared to the original benchmark run.
harness (implicit: Lucena Coder, OpenCode, Copilot, Continue, Kilo Code, Pi) Agent Harness
The six competing coding-agent products/tools under test, each wrapping the identical GLM 5.2 model with its own prompting, context-management and tool-calling strategy.

Propositions

explicitLucena Coder achieved the lowest billed cost ($0.011665) and the lowest total token count (34,102, across 8 model calls) of any harness tested on GLM 5.2.conf: high
explicitPi remained relatively cheap by cost but failed to produce a buildable result — it stopped at build errors in all three runs — so its cost figures do not represent a working solution.conf: high
explicitdevelopedGLM 5.2 (versus the prior GLM 5.1 run of this same benchmark) compressed the cost/token spread across harnesses but did not erase it — the heavier harnesses still spent materially more tokens even with aggressive caching.conf: high
extends ←GLM-5.2 is the new leading open weights model on the Artificial Analysis Intelligence Index · p6 — Both are independent, near-contemporaneous measurements of GLM-5.2's token/cost behavior — this article's task-level finding (43k output tokens/task, up from GLM-5.1's 26k) corroborates, at the Intelligence Index level, the Almedra benchmark's harness-level finding that GLM-5.2 compressed but did not erase the cost/token spread versus GLM-5.1.
explicitSending less unnecessary context to the model helps efficiency more than relying on caching does.conf: medium
implicitcorroboratedHarness design, not the underlying model, is the dominant driver of token/cost efficiency differences among coding agents when the model is held constant.conf: medium
refinesAgentics / Tech Things: Tokenmaxxing is dead, long live tokenmaxxing · p8 — extract-tokenmaxxing#p8 claims GLM 5.2 beats Claude on cost-per-outcome (~5x cheaper) holding the task constant. This benchmark holds the model (GLM 5.2) constant instead and finds harness choice alone produces a comparable-magnitude (~5.4x) cost spread among coding agents — refining the model-level cost claim by showing harness design is at least as significant a confound as model choice.
supportsI benchmarked CC's caveman plugin against 'be brief'. The token math is wild. · p7 — extract-cc-caveman-benchmark#p7 claims token management is becoming a first-class developer skill; this benchmark corroborates it from the harness-selection angle, showing a large, reproducible-in-kind cost spread among six coding-agent products running the identical model and task.
refines ←How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks · p5 — Almedra's benchmark (harness varied, model held fixed) concludes harness design is the dominant driver of efficiency differences. This paper (model varied, harness/OpenHands held fixed) finds model identity is an independent, robust driver of efficiency, stable across shared success/failure subsets. Together they show efficiency has (at least) two independent axes -- harness and model -- rather than either being the sole driver.
supports ←Software Design in the Agentic Age: Placing Your Bets · p4 — Retreat consensus that scoping (Bounded Contexts) and harness engineering control token/context cost corroborates Almedra's empirical finding that harness/context-management design, not the underlying model, drives efficiency differences.
implicitA single fixed prompt, repo, and model, run three times with the worst result discarded and the remaining two averaged, is sufficient evidence to rank six different commercial coding-agent products on efficiency.conf: low
explicitThis is the second run of the Almedra benchmark — the original used GLM 5.1 — establishing a repeatable, longitudinal comparison design across model versions.conf: high

Arguments

Lucena Coder is the most token/cost-efficient coding-agent harness for this class of task, on GLM 5.2. strength: low

Explicit premises:

All agents received the identical prompt, repo, and GLM 5.2 model.
OpenRouter logged calls, tokens, cache reads, cost, and time for every run, giving a common, provider-side measurement.
Correctness was verified first (buildable result, verifier checklist) before cost was compared.
Among the agents that passed, Lucena had the lowest cost and the lowest token count.

Implicit premises:

One fixed prompt against one small, fixed React/Vite repo ('the kind of codebase people actually hand to coding agents') is representative of coding-agent work generally.
Three runs, with the worst discarded and the remaining two averaged, is enough to control for run-to-run variance.
OpenRouter's session-log-derived cost and (minute-granular) time data consistently and fairly reflect each harness's real resource use.
Each harness was tested at a fair, comparable default configuration rather than being differentially tuned.
The benchmark publisher's own product (Lucena Coder) winning its own self-run benchmark does not compromise the comparison's neutrality.

Inferences:

From 'lowest tokens/cost in this repeated single-task run' to the general claim 'Lucena wins on token-efficiency.'
From 'task passed and usage was logged by a third-party router' to 'the comparison is fair and its ranking generalizes.'

Narratives

The correctness verifier is described selecting Azahar 1874 and Turia Moonphase to trigger the two-watch comparison panel, then selecting a third watch (Serra GMT) to confirm the oldest selection (Azahar 1874) is replaced rather than the app breaking or ignoring the new selection.
→ Demonstrates that the pass/fail 'Status' column in the results table reflects a scripted functional check against the prompt's specific requirements (replace-oldest-on-third-select), not just 'it compiled.'

Questions addressed

✅ Which coding-agent harness is most token- and cost-efficient on GLM 5.2 for this feature-addition task?
Lucena Coder, at $0.011665 and 34,102 total tokens across 8 model calls — the lowest of the six harnesses tested.
✅ Does upgrading the underlying model (GLM 5.1 to GLM 5.2) close the token-efficiency gap between coding-agent harnesses?
No — it narrows ('compresses') the spread relative to the original GLM 5.1 run, but a substantial gap (roughly 5.4x in cost between the cheapest and priciest passing agents) remains.
⭕ Is a single-task, single-repo, drop-worst-of-three benchmark, run and published by a vendor whose own product wins, methodologically sound enough to generalize a harness-efficiency ranking to other tasks and codebases?

Gaps

incompleteSingle task, single small repo, single prompt: every ranking claim rests on one two-watch-comparison feature added to one one-page React/Vite site. There is no repetition across different codebases, languages, or task types (bug fix, refactor, larger feature, ambiguous spec), so it is unclear whether the harness ranking observed here — Lucena cheapest, Kilo Code priciest — generalizes at all beyond this exact task shape. This is the central 'rigorous adopter' objection: a coding-agent leader deciding harness spend on org-wide tooling would need this repeated across a representative task suite, not one fixture.
Read next: Look for multi-task, multi-repo coding-agent benchmarks (e.g. SWE-bench-style suites) that report harness-level (not just model-level) cost/token breakdowns across varied task types.
incompleteUndocumented statistical procedure: 'ran each benchmark three times, threw out the worst result, and averaged the remaining two' is a non-standard method with no stated definition of 'worst' (cost? tokens? time? a composite?), no variance or confidence interval reported, and an effective N of 2 per data point once the discard is applied. Readers cannot tell whether the ~5.4x cost spread between agents is a stable effect or noise from a 2-sample average.
Read next: Compare to standard eval-reporting practice (report all runs, report variance/stddev, pre-register the discard rule) and to extract-cc-caveman-benchmark#g2, which flags the identical 'broad-strokes methodology, no raw numbers, no statistical measures' gap in a separate token-efficiency benchmark — this looks like a recurring genre problem in vendor-published agent benchmarks.
incompleteConflict of interest is undisclosed and undiscussed: the benchmark is published by Lucena.One, and its own product, 'Lucena Coder,' is the declared winner on both cost and token count. The piece never addresses how this affects the benchmark's neutrality, what independent verification exists, or why a reader should trust a vendor's self-run comparison of its own tool against five competitors.
Read next: Look for independent (non-vendor-run) replications of this benchmark, or third-party coding-agent efficiency comparisons with disclosed methodology and no product at stake for the publisher.
incompleteCorrectness is measured only as binary build-pass/fail plus a scripted functional check (selection/replacement/panel-contents); none of the prompt's qualitative requirements — 'keep the existing editorial visual direction,' code readability, maintainability of the generated comparison feature, adherence to the codebase's existing patterns — are scored, even though they were explicit requirements in the prompt shown. A harness could satisfy the verifier while producing code a human reviewer would reject.
Read next: Look for coding-agent benchmarks that combine automated pass/fail with human or LLM-judge code-quality scoring (readability, consistency, diff size) rather than functional correctness alone.
incompleteModel-harness confound is acknowledged only narrowly: the piece notes GLM 5.2 'compressed' the field relative to GLM 5.1, implying the model itself matters to the spread, but never disentangles how much of any given harness's efficiency is intrinsic to its context-management strategy versus how well that strategy happens to interact with this specific model. The ranking is never tested against a second underlying model to see if it holds.
Read next: Look for the same harness set benchmarked across multiple underlying models (e.g. Claude, GPT, GLM, DeepSeek) to see whether the harness ranking is model-invariant — this is the natural next test given extract-tokenmaxxing#p8's separate claim about GLM 5.2's cost advantage over Claude at the model level.
closed byHow Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks — The gap asks whether a fixed harness's efficiency ranking would hold across multiple underlying models. This paper runs the reverse but complementary experiment -- one fixed harness (OpenHands), eight different models -- and finds the model-level efficiency ranking is stable across shared-success and shared-failure task subsets, i.e. model identity is an independent driver of efficiency separate from task difficulty, partially answering the model-vs-harness disentanglement the gap calls for (though it does not itself vary harness, leaving the harness-invariance direction still open).
illogicalThe article's own bookkeeping disagrees with itself about whether Copilot passed. The Result table marks Copilot 'Pass,' but 'What We Verified' lists only 'Lucena, OpenCode, Kilo Code, and Continue' as producing a buildable result (Copilot silently omitted), and 'What This Test Tells Us' again names only 'OpenCode, Continue, and Kilo Code' as having 'shipped usable results' (both Copilot and Lucena omitted from that sentence, despite Lucena being the declared cost/token winner). For a benchmark whose entire authority rests on careful, auditable bookkeeping, an internal contradiction about a basic pass/fail fact undercuts trust in the rest of the ledger.
Read next: Flag this as a data-quality/proofreading issue to the publisher; treat all narrative (non-table) claims in this piece as needing cross-check against the raw table before citing them.