Downstream #1: Where Do Our Tokens Go?

Johannes Gehrke · article · 2026-08-01 source ↗ Reader 🖥️

Agent Harness Agent Observability AI Cost Visibility & FinOps AI Unit Economics Capability Provisioning (right-sizing to the task)Context Engineering Evals Open-Weight Models

The cost of agentic coding is set less by which model you use than by how the harness manages context, exploration, and stopping — so engineering leaders should invest in harness discipline, per-workflow model evaluation, and cost visibility rather than wait for cheaper tokens to fix agent economics.

practical · mixed — Opens with a narrative anecdote, summarizes a research paper's empirical findings (theoretical/expository core), then pivots to argumentative/practical guidance for engineering leaders ('what we are doing about them at Uber', closing with three action-forcing questions). The propositions ultimately serve 'here is how you should manage agentic-coding economics', so it reads as practical despite the research summary at its center.

Structure

Part	Role
Opening anecdote: an agent stalls mid-task, is revived by a colleague's joke ('keep trying, I believe in you'), and a Slack thread asks whether the cause was the harness or the model.	narrative hook
Framing of the 'Downstream' newsletter itself: upstream is model progress and headlines, downstream is where consequences land in how software gets built and organizations spend money.	context / scope-setting
Summary of Bai et al., 'How Do AI Agents Spend Your Money?' — four findings from ~16,000 agent trajectories across 8 models on all 500 SWE-bench Verified tasks.	evidence (theoretical core)
'What this looks like in practice at Uber': context-reduction techniques, changed defaults, model choice evaluated on real workflows, internal evals via shadow comparison, and cost-visibility dashboards.	case study / practical application
'What I take away' synthesis: falling token prices won't solve agent economics on their own; the goal is more good software per dollar of intelligence.	conclusion / restatement of unity
Closing: three questions for technology leaders (per-workflow cost visibility, internal evals, AI FinOps ownership) plus a call for reader responses.	call to action

Key terms

Downstream
The organizing metaphor for the newsletter: 'upstream' is frontier-model progress and the headlines that follow each release; 'downstream' is where those changes actually land — how software gets built, how organizations spend money, how companies transform, how people work. The article's subject matter is explicitly downstream, not model capability itself.
harness Agent Harness
The scaffolding wrapped around a model: the code that manages context, runs tools, and decides when to stop. Treated as a first-class variable, on par with or more important than the model itself, in determining both behavior (why an agent stalls) and cost.
good software per dollar of intelligence
The author's replacement goal metric for agentic-coding spend: not the smallest token bill, and not raw token throughput, but software output value normalized by AI spend — explicitly contrasted with 'the goal is not the smallest possible token bill.'
AI FinOps AI Cost Visibility & FinOps
A named, accountable ownership function for AI/agent spend, modeled on cloud FinOps — someone who owns visibility, allocation and cost decisions for AI usage the way someone owns cloud spend today. Posed as a rhetorical question ('does anyone own AI FinOps'), not described as something Uber has already implemented.

Propositions

explicitThe cost of coding agents is driven not only by the task or the model, but by randomness and by the harness — the scaffolding that manages context, runs tools, and decides when to stop.conf: high
extendsAgent Harness Engineering · p1 — Bai et al.'s empirical data (16,000 trajectories, 8 models, one harness) provides quantitative corroboration for the qualitative claim that a great harness matters more than a great model.
explicitToken usage for the same model on the same task can differ by as much as 30x across repeated attempts, so cost should be understood as a distribution rather than a fixed number.conf: high
explicitdevelopedMore tokens do not necessarily produce better results: accuracy improves up to a moderate usage level, then saturates and can decline, because the most expensive runs are usually unproductive loops (repeated file reads, undone edits, circling on a wrong approach) rather than deeper reasoning.conf: high
refinesFragments: July 6 · p7 — Fowler proposes raw token count as an architecture-quality proxy; Bai et al.'s finding (cited here) that identical model+task runs vary up to 30x in cost due to harness and randomness shows the proxy is invalid unless harness and stochastic variance are held constant first — this closes that document's own flagged gap (g1).
extends ←The tokenmaxxing party is over, and Revenium is mopping up · p5 — Extends the single-agent 'unproductive loop' waste pattern (repeated file reads, circling on a wrong approach) to the multi-agent level: circular dependencies between agents as a higher-order version of the same wasted-loop phenomenon.
explicitHuman difficulty estimates are poor predictors of agent cost, because agents and humans have different cost surfaces — a task that looks easy to a person can require extensive exploration for an agent, and vice versa.conf: high
explicitAgents cannot reliably predict their own token usage before starting a task — self-estimates correlate only weakly to moderately with actual usage and tend to underestimate, especially input tokens.conf: high
explicitThere is no universally 'best' model — only the best model for a given workflow — so model choice should be evaluated as a complete workflow measured on real production work, not the model in isolation.conf: high
supportsDon't train the model, evolve the harness · p2 — Both converge on treating model and harness as a coupled unit of evaluation rather than judging a model in isolation.
supportsAgentics / Tech Things: Tokenmaxxing is dead, long live tokenmaxxing · p8 — Uber's own workflow migration to an open-weight model with cost falling and quality holding is a concrete corroborating case for the open-vs-frontier cost-per-outcome finding.
explicitdevelopedPublic benchmarks, even good ones built from real work like SWE-bench, are insufficient for engineering decisions — organizations need their own evals, built as shadow comparisons of a candidate model or harness against an incumbent on the same production work.conf: high
supportsAI models can disappear overnight. Is your engineering team built to survive it? · p11 — Both argue industry benchmarks are useful only for orientation and that organizations need their own evaluation built on their own use cases.
supportsPodcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook · p2 — Convergent claim that the highest-value evals are written against an organization's own data/workflows, not generic benchmarks; Uber's shadow-comparison methodology is a concrete instance.
extends ←Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation · p8 — Where the other document argues organizations need their own evals built as shadow comparisons rather than relying on public benchmarks, this talk supplies a concrete mechanism for building those organization-specific evals: simulation-grounded, verifiable scenarios.
explicitMaking agent cost visible to engineers and managers (real-time spend data, dashboards, targeted notifications) changes usage behavior on its own, without requiring policing.conf: high
explicitFalling token prices alone will not solve the economics of agentic coding, because cheaper tokens can just as easily lead agents to consume more of them without moving outcomes.conf: high
implicitcorroboratedMost engineering organizations currently lack real accountability for AI cost — they don't know their highest-volume AI workflow costs per-workflow (only in aggregate), and no one owns AI cost the way someone owns cloud spend.conf: medium
supports ←The tokenmaxxing party is over, and Revenium is mopping up · p4 — Revenium's 'analysis gap' (teams overwhelmed by usage data, lacking actionable remediation) restates, from a vendor-product angle, the same finding that organizations lack real per-workflow AI cost accountability.
implicitcorroboratedCoding-agent spend has become structurally significant enough — 'one of the fastest-growing lines in the technology budget' — that it now requires the same operational discipline (defaults, gating, dedicated ownership) that organizations already apply to cloud infrastructure cost.conf: medium
supports ←The tokenmaxxing party is over, and Revenium is mopping up · p1 — Independent (vendor-side) corroboration that uncontrolled AI/token spend has grown large enough to require the same operational discipline organizations already apply to cloud infrastructure cost — mirrored here at the whole-enterprise level rather than just coding agents.
implicitTreating model selection as a standalone decision (frontier vs. cheap) is a category error — the model, harness, context architecture and execution environment must be evaluated and chosen together as one coupled system.conf: medium

Arguments

Organizations should evaluate and choose models per-workflow using their own production-work evals, rather than defaulting to the frontier model. strength: high

Explicit premises:

Public benchmarks are built from someone else's work, not yours
Uber moved a high-volume automated code-review workflow from a frontier model to an open-weight model; daily cost fell significantly and review quality held up on a head-to-head comparison against the prior model on the same changes
Harvey and Fireworks reached frontier-level quality at far lower cost with open models on legal-agent benchmarks, and Factory found similar economics in its coding agents

Implicit premises:

Quality parity demonstrated on your own workload is a legitimate substitute for assumed frontier-model superiority
The ongoing cost of running shadow-eval comparisons is smaller than the savings unlocked by picking a cheaper-but-sufficient model

Inferences:

Most organizations currently overpay for AI by defaulting to frontier models out of assumed necessity rather than measured need

The harness — not model choice — is the primary lever for controlling agentic-coding cost, so cost-control efforts should focus on context engineering and stop/loop discipline before or alongside model selection. strength: high

Explicit premises:

Bai et al. found the harness dominates cost variance more than task or model alone across 16,000 trajectories
Input tokens exceed output tokens by roughly two orders of magnitude in every phase of an agent run — agents spend most of their tokens reading, not writing
Uber's context-reduction techniques (deferred tool discovery, keeping large tool responses out of the ongoing prompt, code-oriented tool interfaces, a one-hour prompt-cache duration) directly target this input-token imbalance

Implicit premises:

Most of the context/tool-response volume carried through a session is removable waste rather than necessary grounding — trimming it doesn't cost correctness
The correlation between input-token dominance and cost reflects a causal, fixable inefficiency, not just a description of how LLM agents inherently work

Inferences:

An organization that optimizes only model choice while ignoring harness/context design is optimizing the wrong lever first and will see limited savings

Every technology organization needs a named owner for AI cost visibility and internal evals ('AI FinOps'), analogous to cloud-spend ownership.

Explicit premises:

Visibility changes behavior once engineers can see what their own sessions cost
Most organizations lack per-workflow cost knowledge, only aggregate figures
Public benchmarks alone cannot answer whether a candidate model or harness change is safe for a given production workflow

Implicit premises:

Cost/quality tradeoffs in agentic coding are consequential and fast-changing enough to need a dedicated, ongoing owner rather than periodic review

Inferences:

Organizations without an AI-FinOps-equivalent function will systematically overspend or underinvest, with no way to tell which

Narratives

An engineer's agent stalls mid-task with items still on its to-do list; a colleague's fix is literally 'keep trying, I believe in you' — which works. A Slack thread then debates whether the cause was the harness (a missing stop-and-resume hook) or the model, noting that a different model in the same harness on the same work never stalled at all.
→ Dramatizes, before any data is presented, that unexplained agent behavior is a harness-vs-model diagnostic question — setting up the paper's central finding that the harness (not the model or task alone) is a dominant driver of agent cost and behavior.
Uber moved one of its high-volume automated code-review workflows from a frontier model to an open-weight model; daily cost fell significantly, the workflow stayed the same for developers, and review quality held up when compared against the previous model's findings on the same set of changes.
→ Concrete case supporting the proposition that model choice should be evaluated per-workflow on real work rather than by assumed frontier-model superiority; also models the 'shadow comparison' eval methodology the article recommends generally.
Brief mentions that Harvey and Fireworks reached frontier-level quality at far lower cost with open models on legal-agent benchmarks, and that Factory found similar economics in its coding agents.
→ Extends Uber's single internal case into a claimed industry pattern, guarding (thinly) against the criticism that the open-vs-frontier finding is just one company's anecdote.

Questions addressed

✅ What actually drives the cost of running coding agents?
Task and model matter, but randomness and the harness (context management, tool execution, stop decisions) are central and often dominant drivers.
✅ Does spending more tokens produce better agent results?
No, beyond a moderate point — accuracy saturates and can decline, because expensive runs are usually unproductive loops rather than deeper reasoning.
✅ Can we predict how much a coding task will cost before running it — either by human difficulty judgment or by asking the agent itself?
No to both: human difficulty ratings are weak predictors of agent cost, and agents' own pre-execution estimates correlate only weakly to moderately with actual usage and tend to underestimate.
⭕ Which model should an organization use for agentic coding?
Answered only at the level of a principle ('no universal best, evaluate per workflow on real work'); no decision procedure or threshold is given for when the eval investment required to apply that principle is worth it.
⭕ How does a large engineering organization make coding-agent cost governable in practice?
Uber's concrete levers are listed (context-reduction techniques, changed defaults, shadow evals, cost dashboards) but not assembled into a full operating model, ownership structure, or target metrics.
⭕ Does this whole framework — built at Uber's scale, with a custom open-source harness (OpenCode) and dedicated eval infrastructure — generalize to organizations without comparable engineering resources?

Gaps

incompleteThe piece never states how much internal-eval infrastructure (the 'shadow comparison' methodology it recommends) costs to build and run, so 'build your own evals' is left as an unweighted recommendation — the same critique the article levels at public benchmarks (built from someone else's work, so of limited standalone value) is not reflexively applied to the cost of building your own.
Read next: Read against extract-tiger-teams-evals-agents-ai-engineering-playbook and extract-bockeler-harness-engineering for discussion of eval/harness build cost.
incompleteAll of Uber's reported improvements (cost falling, quality 'holding up', defaults changed) are self-reported, single-organization claims without a controlled comparison, independent replication, or disclosed methodology for the review-quality comparison — the same caveat the article explicitly applies to Bai et al.'s single-benchmark, single-harness scope is not applied to its own case study.
Read next: Read against pieces that quantify harness/model ROI with disclosed methodology (e.g. extract-almedra-token-efficiency-benchmark-glm-5-2) for contrast in rigor.
incompleteWhether the cost-distribution and saturation-curve findings hold outside SWE-bench-shaped work (real GitHub issues with test-based evaluation) — e.g. greenfield feature work, non-Python codebases, or open-ended exploratory design tasks — is not explored, despite the article's own caveat that the numbers come from 'one benchmark and one harness' and 'shapes' rather than exact values should transfer.
Read next: Read alongside broader harness-benchmark literature across task types for cross-domain generalizability.
incompleteThe 'no universally best model, just the best model for a workflow' principle has no stated decision threshold — how much workflow volume or stability justifies the investment in a shadow-eval and model-switching process, versus simply staying on a known-good frontier model for lower-volume or higher-stakes workflows.
Read next: Read against extract-stop-overengineering-your-agent-harness (which raises the analogous unweighted over/under-building question for harnesses) applied instead to model-choice investment.
incompleteThe piece recommends real-time cost visibility and dashboards as a behavior-change lever, but does not address the failure mode of visibility metrics being gamed, cost-consciousness suppressing exploration that would have paid off, or who is accountable if switching to a cheaper model silently degrades quality on an edge case the shadow comparison didn't cover.
Read next: Read against corpus material on tokenmaxxing and metric gaming for the failure modes of cost/usage visibility as a governance lever.

Closes gaps elsewhere

Designing the AI-native engineering organization · g2 — The 'what this looks like in practice at Uber' section gives concrete architectural guidance for AI-cost-awareness — deferring tool discovery, keeping large tool responses out of the prompt, code-execution-style tool interfaces, longer prompt caching, and smaller-context/mid-tier-model defaults — directly the token-efficient design pattern guidance that gap identifies as absent.

Downstream #1: Where Do Our Tokens Go?

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Closes gaps elsewhere

Related documents