The Tokens You Can't Wait For

Shreshta Shyamsundar and Anmol Jain · rss · 2026-07-20 Reader 🖥️

AI Unit Economics Compute & Energy Infrastructure Data Sovereignty Diffusion Language Models (Text Diffusion)Inference Optimization Infrastructure Efficiency Vendor Self-Reported Metrics

Because standard autoregressive LLM decoding is memory-bandwidth-bound and only becomes cost-efficient when batched, enterprises that bought owned/sovereign GPU compute for latency-bound, single-request workloads are paying for chronically idle hardware — and text diffusion, which is compute-bound even at batch size one, is the precise fix for exactly that mismatch, not a general-purpose replacement for autoregressive models.

practical · mixed — Argumentative and expository together: the piece builds a mechanistic technical explanation (memory-bound vs. compute-bound inference, arithmetic intensity) in service of a practical routing recommendation — which architecture (owned diffusion, owned autoregressive, or external API) to run which workload on. It ends in an explicit prescription, not just an account of how things work.

Structure

Part	Role
The Singapore bank vignette: eight H100s bought for sovereignty/lock-in reasons, sitting idle most of the night while the finance team questions the spend.	context / motivating narrative
The physics of autoregressive decoding: weights must be re-streamed from HBM through compute units for every token, giving arithmetic intensity near 1 at batch size one; batching is the only escape hatch and can swing cost per token 10-30x.	mechanistic evidence
The latency-bound vs. batchable distinction: overnight queues are trivially batchable, but sub-second single requests (code completion, onboarding checks) cannot accumulate concurrency and stay memory-bound regardless of total volume.	mechanistic evidence
Text diffusion's mechanism: parallel denoising of a full token block gives compute-bound behavior (arithmetic intensity in the hundreds) even at batch size one, with cited throughput figures from Mercury, Mercury 2, Gemini Diffusion, LLaDA.	evidence for the proposed fix
Fair-comparison correction: speculative decoding (Medusa, EAGLE) and MoE already give autoregressive serving 2-4x single-stream gains and reduced memory movement, so diffusion must be compared against an optimized baseline, not a naive one.	counter-consideration / qualification
The unit-economics formula (effective cost per token = node cost per hour / (throughput x utilization)) and concrete AWS p5 vs. API pricing comparison.	quantitative evidence
The bank's two document-processing workloads (overnight KYC batch vs. real-time onboarding/LC clearing) as a worked contrast showing where architecture choice matters and where it doesn't.	case-study evidence
The generalized routing rule: batchability x per-token value determines whether work should run on owned diffusion, owned autoregressive, or stay on frontier APIs.	conclusion / practical prescription
Diffusion's real constraints: 85-95% of autoregressive quality, immature tooling (roughly where autoregressive serving was in early 2024), a moving baseline as speculative decoding/MoE keep improving, and prices that aren't fixed.	qualification / limits
Closing reframe: the 'hangover' isn't bad hardware-buying, it's running sovereignty-motivated hardware at cloud-style concurrency it can't reach; diffusion is a narrow fix for the latency-bound slice, not a hardware-ROI rescue.	conclusion

Key terms

GPU hangover
The state of enterprise-owned GPU compute — bought for sovereignty, data-control, or lock-in-avoidance reasons — sitting mostly idle because the workloads actually run on it are latency-bound and can't reach the concurrency that would justify the hardware's cost.
arithmetic intensity
Operations performed per byte of data moved from memory; used as the diagnostic for whether a workload is memory-bound (near 1, autoregressive at batch size one) or compute-bound (in the hundreds, diffusion or well-batched autoregressive).
memory-bound / compute-bound
Memory-bound: throughput limited by how fast weights can be streamed from HBM to compute units, not by arithmetic capacity. Compute-bound: throughput limited by how much arithmetic the chip can do, with memory feed no longer the bottleneck.
effective cost per token
For owned compute: node cost per hour divided by (throughput times utilization) — the true per-token cost once idle time is accounted for, as opposed to a public API's flat concurrency-independent per-token price.
latency-bound (workload)
A request that must return before enough concurrent requests can accumulate to fill a batch — the defining property that makes batching-based autoregressive economics inapplicable, regardless of total request volume.
sovereign compute
Owned or dedicated GPU clusters purchased so that customer/regulated data and processing never leave the enterprise's own infrastructure — a decision driven by data-control and lock-in-avoidance, independent of and often in tension with token-cost economics.

Propositions

explicitcorroboratedStandard autoregressive LLM decoding is memory-bandwidth bound at low batch sizes because the full weight set must be re-streamed from the GPU's high-bandwidth memory through the compute units for every single token generated.conf: high
supports ←DiffusionGemma: 4x faster text generation · p1 — Independently describes the same memory-bandwidth-bound-at-low-batch-size mechanism for autoregressive decoding that the corpus's systems-level analysis names generically.
explicitBatching amortizes the cost of reading weights across many concurrent requests, and on the same hardware can swing cost per token 10- to 30-fold between small and large batch sizes.conf: high
explicitLatency-bound, single-stream requests (code completion, real-time onboarding checks, a blocked agentic step) cannot be batched because there is no concurrent work to accumulate within the latency budget, and no amount of total request volume changes this.conf: high
explicitdevelopedText diffusion models are compute-bound even at batch size one because each denoising pass does real arithmetic across a whole token block in parallel, giving arithmetic intensity in the hundreds versus roughly 1 for autoregressive decoding.conf: high
extends ←DiffusionGemma: 4x faster text generation · p2 — Supplies a concrete production example (26B MoE, 256-token blocks) of the compute-bound parallel-denoising mechanism the corpus piece analyzes in the abstract.
extends ←Introducing System One Models & Jev · p4 — Generalizes the corpus's non-autoregressive/parallel-decoding speed argument (there applied to diffusion text models trading some accuracy for speed) to a further extreme: giving up free-form string generation entirely in favor of typed, parallel decision outputs.
explicitdevelopedEarly diffusion-model deployments (Inception Labs' Mercury and Mercury 2, Google's Gemini Diffusion, open-source LLaDA) report roughly 1,000+ tokens per second on frontier GPUs, an order of magnitude faster than autoregressive single-stream decode.conf: medium
extends ←DiffusionGemma: 4x faster text generation · p3 — Adds DiffusionGemma's concrete throughput figures (1000+ tok/s H100, 700+ tok/s RTX 5090) to the corpus's running list of diffusion deployments reporting roughly 1,000+ tok/s on frontier GPUs.
explicitdevelopedSpeculative decoding and its descendants (Medusa, EAGLE), plus mixture-of-experts routing, already give autoregressive serving roughly 2- to 4-fold single-stream speedups with no quality loss, so any diffusion-vs-autoregressive comparison must use this optimized baseline, not a naive one.conf: high
extendsGuest post: AI Inference Is Breaking Unit Economics · p4 — Where Yandex's stacked optimizations (quantization + EAGLE3 speculative decoding + KV cache reuse) squeeze more throughput out of autoregressive decoding, this piece adds a structurally different lever — diffusion's parallel denoising — for the specific case (hard single-stream latency budget) where stacking incremental autoregressive tricks still isn't enough.
extends ←DiffusionGemma: 4x faster text generation · p7 — The corpus piece's point — that a fair diffusion-vs-autoregressive comparison must use an optimized (speculative-decoding/MoE) autoregressive baseline, not a naive one — is exactly the missing disclosure this document's '4x faster' claim leaves open.
extends ←Transcript for Cursor Team: Future of Programming with AI | Lex Fridman Podcast #447 · p6 — Cursor's speculative-edits technique (feeding back existing code as a 'draft' for the model to regenerate, slowing only at points of disagreement) is a code-specific precursor to the speculative-decoding-plus-MoE serving stack the later piece describes as now standard practice.
explicitThe effective cost per token of owned compute equals node cost per hour divided by (throughput times utilization); a public API instead prices per token with no idle penalty regardless of concurrency.conf: high
explicitcorroboratedDiffusion trades accuracy for speed, landing at roughly 85-95% of strong autoregressive baselines — competitive on structured output extraction but trailing by 5-15% on hard reasoning tasks.conf: medium
supports ←DiffusionGemma: 4x faster text generation · p5 — Confirms the general 'diffusion trades accuracy for speed' finding, though without the quantified 85-95%-of-autoregressive-quality figure the corpus piece provides for other diffusion models — the gap this extraction flags.
explicitEnterprises bought sovereign/owned GPU clusters for reasons unrelated to token economics (data-control discomfort, avoidance of vendor lock-in), then ran them expecting cloud-style concurrency that their most valuable internal workloads — the latency-bound ones — structurally cannot reach.conf: high
supportsLocal AI · p2 — The bank's own sovereignty/lock-in-avoidance rationale for buying GPUs is a concrete, sector-specific instance of the regulation- and geopolitics-driven momentum toward local/owned AI that Loukides describes at the industry level.
implicitcorroboratedThe architecture-routing decision an enterprise should make is not 'diffusion versus autoregressive' in the abstract but a two-axis test — whether the work can be batched, and how much each token is worth — that sorts workloads into owned diffusion, frontier autoregressive, or whatever already runs offline batch efficiently.conf: high
extendsYour Company Needs an Energy Strategy for AI's Next Phase · p7 — Sharpens the generic 'firms over-route to frontier models that exceed the work's requirements' critique into a mechanistic two-axis rule — route by batchability and per-token value, not just perceived task difficulty — that specifies exactly which workloads belong on a smaller/different model versus a frontier one.
supports ←DiffusionGemma: 4x faster text generation · p4 — Google's own guidance — good for local/low-concurrency, poor fit for high-QPS cloud batching — independently confirms the batchability axis of the routing framework proposed there.
implicitAny cost or routing conclusion in this analysis is only as durable as two prices that move independently of the architecture argument — the API rate compared against and the hardware rate negotiated — so the routing rule needs periodic re-evaluation, not one-time adoption.conf: medium
implicitBecause diffusion's main current handicap (immature serving tooling, roughly where open-source autoregressive serving stood in early 2024) is an engineering maturity gap rather than a theoretical ceiling, the case for diffusion in its sweet spot should strengthen over time rather than weaken.conf: medium

Arguments

Text diffusion is the correct architecture specifically for latency-bound, decode-heavy, sovereignty-constrained enterprise workloads — not a general-purpose replacement for autoregressive models. strength: high

Explicit premises:

Autoregressive decoding is memory-bound at low concurrency (arithmetic intensity ~1 at batch size one).
Batching is the only fix for that bottleneck, and batching is structurally unavailable under a hard sub-second latency budget.
Diffusion is compute-bound even at batch size one (arithmetic intensity in the hundreds), because it does real arithmetic across a full token block per denoising pass.
Early real deployments report diffusion delivering roughly an order-of-magnitude single-stream throughput advantage over autoregressive decode.

Implicit premises:

The enterprise workloads that matter for this argument (code completion, onboarding checks, agentic steps) are representative of what sovereign-compute buyers actually run, not edge cases.
Owned GPU cost is dominated by hourly node pricing rather than per-token metering, so utilization — not raw hardware capability — is the lever that determines whether the purchase was worth it at all.
Vendor-reported throughput and quality figures (1,100 tok/s, 85-95% quality) are directionally trustworthy enough to build a routing recommendation on, even though the authors simultaneously caution that these exact figures need independent verification against the reader's own data.

Inferences:

If diffusion serving infrastructure matures at the pace autoregressive serving did (2024 to 2026), the 'immature tooling' caveat may retire within roughly two years, removing the main non-technical friction against adoption.
Conversely, if speculative decoding and MoE keep narrowing the autoregressive gap faster than diffusion tooling matures, diffusion's addressable share of workloads could shrink even as its raw throughput advantage persists — the article does not weigh which trend is currently winning.

Sovereign/owned GPU clusters bought for data-control reasons are not economically justified against public APIs on cost alone, except in two narrow situations: expensive frontier/reasoning tokens, or workloads whose data legally cannot leave the building. strength: medium