Baseten CEO Tuhin Srivastava on the AI Inference Crunch, Custom Models

No Priors (Sarah Guo, Elad Gil) · podcast · 2026-05-09 Reader 🖥️

AI Unit Economics Compute & Energy Infrastructure Jevons Paradox Moats & Defensibility Open-Weight Models

AI inference is the enduring, last-standing market because intelligence only begets more intelligence, and the companies that win it will combine software excellence with strategic control of scarce compute supply.

practical · mixed — A CEO interview mixing market analysis (theoretical) with strategic prescriptions for how companies should approach inference, custom models, and scaling (practical). The conversation is argumentative in structure — hosts probe, guest defends positions.

Structure

Part	Role
Baseten's 30x growth and what's driving it	Evidence — establishes that the market is real and accelerating
The independent application layer will persist	Argument — defends the existence of a customer base beyond the labs
Open source models have crossed a capability chasm; origin doesn't matter	Context — explains why custom/open-source inference demand is exploding
Post-training and inference are two sides of the same problem	Core thesis component — the inference-post-training loop is the product
The supply crunch is worse than people think	Evidence — grounds the 'last market' claim in physical scarcity
Nvidia's supply chain and ecosystem moat	Analysis — explains why multi-chip diversification is hard near-term
Jevons paradox: cheaper inference begets more inference	Core argument — the demand curve never saturates
Scaling the company: leadership, ops culture, hiring philosophy	Narrative — illustrates what it takes to operate at this scale
The future: concierge everything, extinction for non-adopters	Conclusion — the long-term vision

Key terms

inference cloud Compute & Energy Infrastructure
A unified runtime fabric spanning many physical clouds, optimized specifically for serving AI model inference at scale — not just GPU-as-a-service
the last market Compute & Energy Infrastructure
Even if AGI arrives, all remaining compute work reduces to inference; it is the terminal market that subsumes everything else
custom models
Not fine-tuned from scratch but open-source base models modified via post-training, quantization, or compilation for a specific use case — 95%+ of Baseten's tokens
post-training
RL-based and other techniques applied after base model training to specialize a model on a company's proprietary reward signal/user data
user signal Moats & Defensibility
Proprietary data generated by end-users interacting with a product (e.g., clinician edits in Abridge) that only that company can access — the moat for the application layer
runtime fabric Compute & Energy Infrastructure
Baseten's abstraction layer across 18 clouds / 90 clusters that enables failover, latency optimization, and rapid onboarding of new compute providers

Propositions

explicit95%+ of tokens served on Baseten come from custom (post-trained or compiled) models, not vanilla open-source weightsconf: high
explicit⚡ disputed & corroboratedThe independent application layer will survive against the labs because companies possess unique user signal that labs cannot accessconf: high
supports ←Travel Through the Lens of AI with Booking.com CEO Glenn Fogel · p12 — Fogel's claim that supply connectivity and listings an AI-native entrant cannot easily obtain protect the incumbent parallels baseten's thesis that the independent application layer survives against the labs via unique assets/user signal the labs cannot access.
contradicts ←Is Your AI Training the Competition? · p9 — Baseten's CEO argues the application layer survives because companies hold 'unique user signal that labs cannot access'; this article argues labs effectively do access it, indirectly, via the feedback/flywheel/distillation channels it documents — the signal is not as inaccessible as claimed.
explicitBy count of inference, 99% of the enterprise market has not yet come online for AIconf: high
explicitPost-training and inference are both sides of the same problem — tightly coupled, not separate concernsconf: high
explicitdevelopedThe compute supply crunch is worse than the public narrative suggests, and is compounded by an operational quality crunch among providersconf: high
refines ←Presentation: From Fab To Token - The State Of The Market · p1 — Identifies TSMC wafer-fabrication capacity specifically as the mechanism behind the 'compute supply crunch worse than the public narrative suggests' that Baseten's CEO describes more generally.
explicitGPU-as-a-service without a software layer is commodity and not sticky; inference with a software layer is extremely stickyconf: high
explicitIn a world of constrained compute, owning the compute itself — not just the software — is a strategic assetconf: high
explicitdevelopedNvidia's supply chain, CUDA ecosystem, and developer tooling make it nearly impossible for alternative chips to compete in the next couple of yearsconf: high
extends ←Presentation: From Fab To Token - The State Of The Market · p4 — Baseten's CEO attributes NVIDIA's dominance to its CUDA ecosystem/developer tooling moat; Nanos adds a separate, upstream mechanism — NVIDIA's dominant share of TSMC's 3nm wafer allocation — that caps competitors regardless of software ecosystem effects.
explicitestablishedDevelopers insert more intelligence when inference gets cheaper — Jevons paradox applies to AI inferenceconf: high
supports ←Presentation: A Few Predicted Talks From QConAI 2030 · p3 — Independent convergence: both speakers (competing inference-platform founders) separately name Jevons paradox as the reason cheaper tokens produce more, not less, total spend.
supports ←Howard Yu's Post — American AI models went from 70% of usage to 30% in 12 months · p3 — The agentic multiplier is the Jevons mechanism — cheaper inference pulls in far more calls, which is exactly why per-call price becomes decisive.
explicitcorroboratedChinese open-source model subsidies effectively constitute a subsidy to US enterprises adopting those modelsconf: high
supports ←Howard Yu's Post — American AI models went from 70% of usage to 30% in 12 months · p6 — US firms adopting Chinese open models: Baseten frames Chinese open-source subsidies as an effective subsidy to US enterprises — the same adoption surge Casado quantifies.
explicitcorroboratedThe H100 is 4.5 years old, its price is still rising, and it may have a useful life of 9 yearsconf: medium
supports ←Andreessen Horowitz's Post · p9 — Independent corroboration of the same specific fact: H100 pricing is still rising despite the hardware aging, which both sources read as evidence that demand is outpacing supply.
explicitNo post-training pre-product-market-fit — companies should prove value with frontier models first, then optimize with custom modelsconf: high
explicitcorroboratedAI will not reduce the number of software engineers but will increase the total amount of software builtconf: medium
supports ←Arvind Narayanan on Why AI Isn't All That Revolutionary · p3 — Both claim AI writing most/all code does not reduce software-engineer headcount — Narayanan explains the mechanism (decide/deliver layers absorb the saved time), the Baseten CEO states the same empirical outcome (more software built, not fewer engineers).
explicitCompanies that do not embrace AI face extinctionconf: high
implicitThe frontier closed-source models (Anthropic, OpenAI, Google) remain the most capable — open source has not overtaken themconf: high
implicitInference companies will need to go public sooner than typical SaaS companies because of the capital requirements of long-term compute contractsconf: medium

Arguments

The application layer will survive against the frontier labs strength: high

Explicit premises:

Companies have unique user signal that labs cannot access
Proprietary user signal enables post-training on unique reward functions
Workflows (not just model calls) encode company-specific value

Implicit premises:

Labs will not acquire or replicate the distribution needed to gather domain-specific user signal at scale
Regulatory and data-access barriers will persist, especially in healthcare

Inferences:

The value accrues to companies closest to the user, not closest to the weights

Inference is 'the last market' — it cannot be competed away strength: high

Explicit premises:

Even with AGI, everything reduces to inference
Jevons paradox means cheaper inference drives more demand, not less
Customers never reach 'enough intelligence' — they always want better answers

Implicit premises:

Training will eventually become a solved/commoditized upstream step while inference remains the ongoing operational cost
The intelligence demand curve has no natural ceiling

Inferences:

Inference companies may have more durable moats than training/lab companies

Owning compute is the dominant competitive strategy in inference strength: high

Explicit premises:

Supply crunch is severe and worsening
Quality suppliers are a small subset — maybe 3-4 'gold tier' clouds
Long-term contracts now require 3-5 year terms with 20% TCV prepaid
Low cost of capital enables compute acquisition which enables demand capture

Implicit premises:

The supply crunch will persist long enough for compute-owning companies to build durable advantages
Demand growth will outpace supply growth for the foreseeable future

Inferences:

This creates a capital-intensive dynamic more like energy/telecom than traditional SaaS, favoring public companies or well-capitalized private ones

The US must develop strong domestic open-source models strength: medium

Explicit premises:

Five Chinese labs are producing open-source models; the US struggles to match
If the US loses access to cheap open-source intelligence, innovation slows
DeepSeek runs at ~20% the cost of comparable models with similar quality

Implicit premises:

Geopolitical tensions could restrict access to Chinese models in the future
Open-source models are critical infrastructure, not just nice-to-have

Inferences:

The US open-source AI gap is a national security concern, not just a market concern

Narratives

Abridge (ambient medical scribe) as an example of unique user signal — clinicians' edits to notes and downstream EMR workflows create data only Abridge can access, making it defensible against labs
→ Supports the proposition that the application layer survives because of proprietary user signal
Baseten runs 90 clusters across 18 clouds and can onboard a new provider in half a day; they hold a daily 4pm capacity meeting
→ Supports the proposition that operational excellence and multi-cloud flexibility are key competitive advantages in the supply crunch
Meeting with senior AWS executives where all their pagers went off multiple times in 45 minutes
→ Normalizes the ops-heavy, always-on culture as inherent to infrastructure companies — not a dysfunction but a selection mechanism
Co-founder's 7-year-old asking 'Is that a P0?' when the pager goes off
→ Reinforces the ops-culture narrative with humor — infrastructure is a lifestyle, not just a job
Tuhin once suggested to Sarah Guo that Baseten should hire only engineers instead of salespeople
→ Illustrates his evolution as a founder — from engineering purism to recognizing the necessity of leadership and go-to-market roles
The PaLM acquisition — a customer doing post-training on Baseten realized they'd need to become an inference company; Baseten realized they needed research capability
→ Supports the argument that post-training and inference are converging into one problem
Someone told Tuhin after the DeepSeek moment: 'We should just forget this is a Chinese model and act like it came from Meta'
→ Supports the argument that model origin matters less than model capability and availability

Questions addressed

✅ Will the independent application layer survive against the frontier labs?
Yes — unique user signal and downstream workflows create defensible positions that labs cannot replicate
✅ Should companies use Chinese open-source models despite geopolitical concerns?
Yes — no evidence of embedded threats; the real risk is NOT having access to cheap open-source intelligence
✅ When should companies invest in custom models?
After proving product-market fit with frontier models — 'no post-training pre-PMF'
✅ What creates a dominant player in inference?
Software layer stickiness + compute ownership + operational excellence — not any single factor
⭕ Will alternative chips challenge Nvidia?
Inevitable long-term, but near-term (2-3 years) Nvidia's ecosystem and supply chain are unassailable
✅ Does cheaper inference cannibalize inference revenue?
No — Jevons paradox; developers insert more intelligence, agents run longer, demand expands
⭕ What is the long-term equilibrium of AI adoption?
'Concierge everything' — but specifics of economic and social impact left unexplored

Gaps

incompleteThe '400% NDR' and extreme stickiness claim is stated without discussing what happens when the supply crunch eases. If compute becomes abundant, does the software moat alone sustain that retention, or is current stickiness partly a supply-lock-in artifact?
Read next: Analysis of cloud platform switching costs and lock-in dynamics in prior infrastructure transitions (e.g., CDN market evolution)
incompleteThe Jevons paradox argument is compelling but treats demand as infinitely elastic. No discussion of whether there are diminishing returns to intelligence insertion in specific verticals, or whether regulatory/cost ceilings emerge.
Read next: Research on price elasticity of compute in enterprise AI deployments; historical analogies to electricity or bandwidth consumption curves
closed byAgentics / Tech Things: Tokenmaxxing is dead, long live tokenmaxxing — That gap asked whether there are diminishing returns to intelligence insertion or cost ceilings, versus infinitely elastic demand. This doc adds concrete discussion where there was none — a diminishing-returns data point (AISI's Mythos runs showed none across tested token budgets) and a named cost ceiling (CFO scrutiny driving substitution to cheap open models). Partial only: the data point is security-specific and actually reinforces the elastic-demand assumption rather than settling whether ceilings ultimately exist.
incompleteThe 'last market' framing assumes training commoditizes while inference persists. But if continuous post-training is integral to inference (as Tuhin himself argues), the boundary between training and inference revenue may dissolve, complicating the 'last market' claim.
Read next: Papers or analyses on the convergence of training and inference workloads; Nvidia's own framing of this boundary
uninformedNo discussion of energy constraints, cooling, or sustainability as potential bottlenecks alongside the GPU supply crunch. The physical infrastructure dimension is absent.
Read next: Reporting on data center power constraints, particularly in relation to AI inference workloads
closed byAI's environmental costs threaten water, land and climate — Partial, cross-framing closure: adds the missing physical-infrastructure/sustainability dimension (energy, water, and land constraints on data centres) that gap flagged as absent, though this document frames it as environmental cost to the planet rather than as a bottleneck constraining AI's ability to scale — the angle the original gap asked about.
incompleteThe 'concierge everything' vision is stated as unambiguously positive with no engagement with critiques of AI-mediated services (deskilling, dependency, privacy, equity of access).
Read next: Critical perspectives on AI-mediated services from Shoshana Zuboff, Meredith Whittaker, or similar voices

Baseten CEO Tuhin Srivastava on the AI Inference Crunch, Custom Models

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Related documents