Generative AI in the Real World: Aishwarya Naresh Reganti on Making AI Work in Production

Ben Lorica and Aishwarya Naresh Reganti · podcast · 2026-04-16 source ↗ Reader 🖥️

Agent Harness Agent-Native Design Decision-Making Practices Enterprise AI Adoption & Readiness Evals Forward-Deployed Engineer Human-in-the-Loop AI Workflow Patterns

Shipping generative AI in production requires inverting the traditional software effort profile — spending most resources on post-deployment calibration, data understanding, cross-functional collaboration, and workflow-level design rather than on initial build — because the natural-language interface opens an unbounded error surface that only continuous human-in-the-loop refinement can tame.

practical · mixed — primarily expository with argumentative elements — Conversational interview between a podcast host (Ben Lorica) and a practitioner-consultant (Aishwarya Reganti). Reganti supplies most substantive claims from direct enterprise consulting experience. Lorica steers topics and occasionally offers contrasting framings. The conversation is structured around recurring practitioner pain points rather than a single linear argument.

Structure

Part	Role
The 80-20 flip	Core reframing — AI development inverts the traditional software effort split, requiring 80% of work on calibration after deployment rather than on building
Data-centricity and manual groundwork	Argues that understanding data distributions, workflows, and user behavior through manual effort is the underrated foundation of production AI
Complementary strengths of developer and ML personas	Maps what traditional developers (design thinking, scalable architecture) and ML/data-science practitioners (data intuition, evaluation discipline) each contribute
Evals as a process, not a checkbox	Critiques the hype-driven reduction of evaluation to a buzzword; argues evals must be actionable, iterative, and distinguish model-level from product-level evaluation
Optimization pyramid: effort, performance, cost, latency	Prescribes a sequencing framework — start low-effort to establish feasibility, optimize performance, then address cost and latency
Reliability over accuracy — workflows over autonomy	Argues that predictable, auditable systems (copilot + human approval) are preferable to fully autonomous agents; design decisions should precede building
Human judges vs. LLM judges	Prescribes starting with human judgment; transition to LLM judges only when evaluation rubrics can be codified in natural language from sufficient examples
Enterprise deployment reality	Reports that most enterprise AI is internal/productivity-focused, not customer-facing; customer-facing deployments use phased human-AI triage
Model convergence and the application layer	Argues frontier models are converging on capability, so competitive advantage lies in the harness/application layer rather than model selection
Career advice: agent-native operators	Closing section — advises early-career people to become 'agent-native,' focus on design thinking and end-to-end understanding rather than execution

Key terms

Propositions

explicitAI product development inverts the traditional software effort split: 20% building, 80% calibration.
explicitThe natural-language interface of AI products opens a much larger surface area for errors than deterministic UI workflows.
explicitThe single most important thing anyone building AI products can do is take the time to look at their data.
explicit⚡ disputed & corroboratedYou cannot automate the understanding of your own data by having an LLM evaluate itself — manual human effort is required to set up agents properly.
supports ←AI Companies Are Hiring More · p10 — Ford rehiring 350 veteran engineers to fix and retrain its AI tools is concrete evidence that human expertise cannot be automated out of AI setup.
contradicts ←Kaggle is making AI benchmark creation effortless · p8 — Kaggle assumes AI agents can rigorously author evals, including self-evaluation; Reganti insists LLM self-evaluation cannot replace manual human effort.
explicitTraditional software engineers excel at design thinking, scalable architecture, and treating the model as an API with proper security and system design around it.
explicitThe term 'evals' is overloaded and hyped to the point of confusion, conflating model-level benchmarks with product-level evaluation processes.
explicitEvaluations must be actionable — if you cannot derive a concrete improvement action from an eval result, it is not useful.
explicitWhen building AI products, you should sequence optimization as: low effort first (establish feasibility), then performance, then cost and latency.
explicitPredictable reliability (knowing which 10% you get right) is more valuable than higher but unpredictable accuracy.
explicitcorroboratedIf a task can be solved with a workflow (with human audit points), do not make it autonomous.
supports ←Expedia Uses AI Driven Service Telemetry Analyzer to Accelerate Incident Investigation · p2 — Reganti's principle -- 'If a task can be solved with a workflow (with human audit points), do not make it autonomous' -- is exactly the design choice STAR embodies in production: a fixed telemetry-to-prompt-chain-to-report workflow with mandatory human review before any action, deliberately declining function calling, RAG, memory and autonomous tool use. STAR is a concrete, named, production instance of that abstract principle.
explicitHuman judges should be used first for evaluation; transition to LLM judges only after rubrics can be codified from sufficient human-annotated examples with reasoning.
refinesHarness design for long-running application development · p4 — Reganti conditions criteria-based LLM grading: only after rubrics are codified from sufficient human-annotated examples should LLM judges replace human judges.
explicit70-80% of LevelUp Labs' enterprise engagements are productivity and ops focused, not customer-facing, primarily because of trust and reliability concerns.
explicitestablishedFrontier models are converging on capability; competitive advantage now lies in the harness and application layer rather than model selection.
supports ←Andreessen Horowitz's Post · p16 — Both claim that as frontier model capability converges, competitive advantage migrates to the harness/application/packaging layer rather than the model itself.
supports ←The new GPT-5.6 family: Luna, Terra, Sol · p6 — Hands-on parity between GPT-5.6 Sol and Claude Fable 5 on complex coding is direct evidence that frontier models are converging on capability, per p13.
explicitcorroboratedModel personality differences (not capability differences) are the real barrier to model portability, and these will not converge because vendors treat them as competitive differentiators.
extends ←Andreessen Horowitz's Post · p14 — Both independently argue that model 'personality' (not capability) is the real barrier to treating models as interchangeable — this post frames it as a portfolio requirement, the other as a portability barrier that won't converge because vendors treat it as a differentiator.
supports ←DeepSeek v4, and the end of the OpenAI/Microsoft AGI clause · p12 — OpenAI's own don't-migrate-prompts guidance for GPT-5.5 is direct evidence that model personality differences block portability.
explicitEarly-career people who are 'agent-native' have a genuine hiring advantage because they naturally delegate to and orchestrate AI agents in ways experienced workers do not.
implicitThe valued professional skill is shifting from execution to design and orchestration, because execution is increasingly handled by agents.
explicitCompanies that fail to involve governance and compliance stakeholders early in AI projects end up scrapping entire projects.
explicitCross-functional collaboration must be tighter and more continuous for AI products than for traditional software — the feedback loop must be shared across engineering, SMEs, and testing.

Arguments

AI product development requires fundamentally different effort allocation than traditional software (the 80-20 flip)

Explicit premises:

AI products use natural-language interfaces, which allow users to interact in unbounded ways (P2)
User behavior changes over time, as seen with ChatGPT from 2022 to present
Models themselves change, requiring realignment

Implicit premises:

Traditional software's deterministic interfaces constrain user behavior, limiting the error surface
The cost of post-deployment iteration in AI is inherently higher because errors are harder to anticipate

Inferences:

Looking at your data is the single most important practice for AI product builders

Explicit premises:

People from software backgrounds assume a few test cases are sufficient (P3)
LLMs cannot reliably self-evaluate and self-correct (P4)
Traditional ML practitioners understand this because they have always curated data for training

Implicit premises:

The distribution of real user queries is significantly different from developer test cases
Human understanding of data patterns is not yet replaceable by automated analysis

Inferences:

Prefer workflows with human audit points over fully autonomous agents

Explicit premises:

AI models sometimes generalize in undesirable ways (P9)
Building is cheap today — Claude Code can produce a customer support agent in minutes
One mistake in a customer-facing agent can generate bad PR (P12)

Implicit premises:

The cost of a false positive (wrong autonomous action) exceeds the cost of slower human-in-the-loop processing
Auditability at each decision point is a prerequisite for enterprise trust

Inferences:

T
h
e
a
s
y
m
m
e
t
r
y
b
e
t
w
e
e
n
t
h
e
l
o
w
c
o
s
t
o
f
b
u
i
l
d
i
n
g
a
n
d
t
h
e
h
i
g
h
c
o
s
t
o
f
e
r
r
o
r
s
m
e
a
n
s
d
e
s
i
g
n
t
h
i
n
k
i
n
g
(
w
h
e
r
e
t
o
i
n
s
e
r
t
h
u
m
a
n
j
u
d
g
m
e
n
t
)
m
a
t
t
e
r
s
m
o
r
e
t
h
a
n
b
u
i
l
d
s
p
e
e
d
.

Human judges should precede LLM judges in the evaluation pipeline

Explicit premises:

Humans can often judge quality without being able to articulate their rubrics (the 'viral post' anecdote)
LLM judges need explicit rubrics written in natural language
Rubrics require 100-200 annotated examples with reasoning

Implicit premises:

Tacit knowledge must be made explicit before it can be encoded for an LLM
Premature automation of evaluation produces unreliable signals

Inferences:

Model selection is no longer a primary concern for enterprises

Explicit premises:

Frontier models are reaching feature parity quickly (P13)
Enterprises already have vendor relationships and are comfortable with them
The application/harness layer is where differentiation occurs

Implicit premises:

Switching costs between models are real but manageable for enterprise use cases (functionality > personality)
The rapid convergence of model capability makes model-level differentiation transient

Inferences:

Narratives

Questions addressed

⭕ What separates a real AI product from a good demo?
resolved
⭕ What do non-ML people consistently get wrong when building AI applications?
resolved
⭕ What do traditional software developers bring to the table that ML people lack?
resolved
⭕ Why is the term 'evals' problematic?
resolved
⭕ When should you use an LLM judge versus a human judge?
resolved
⭕ Are enterprises actually deploying foundation models in customer-facing applications?
partially resolved
⭕ Is model portability/neutrality a real concern for enterprises?
resolved
⭕ What career advice applies to early-career people in the AI era?
resolved

Gaps

incompleteThe 80-20 flip is asserted from consulting experience but not quantified. No data is offered on what 'calibration' costs in practice relative to build — the ratio could vary enormously by domain, and the 80-20 framing may be more rhetorical than empirical.
incompleteThe conversation assumes enterprises have the organizational capacity to run tight cross-functional feedback loops, but does not address the structural barriers (incentives, tooling, culture) that prevent most organizations from achieving this.
closed byYour AI strategy has a trust problem, not a tooling problem — Names the structural barriers the gap said were unaddressed: titles as permission structures, expensive-because-slow decision loops, and leadership distrust are why most orgs cannot run the tight cross-functional feedback loops AI products require (p1-p3, p11).
uninformedThe legal and compliance discussion is acknowledged as outside Aishwarya's expertise. Given that governance failures can kill finished projects (as her own anecdote shows), the treatment is surprisingly thin — no frameworks, regulations, or concrete practices are discussed.
illogicalAishwarya argues that model personality differences will not converge because vendors treat them as competitive differentiators, but also argues that model capabilities are converging rapidly. The distinction between 'capability' and 'personality' is never defined rigorously — prompt sensitivity, output style, and error patterns could plausibly converge as capabilities do.
incompleteThe claim that early-career 'agent-native' workers have a hiring advantage is supported only by Aishwarya's personal hiring experience at a small consultancy. Whether this generalizes to larger enterprises with different incentive structures is unexamined.
incompleteThe optimization pyramid (effort -> performance -> cost -> latency) is offered as a sequential framework but no guidance is given on when to stop optimizing performance and move to cost/latency, or how to handle cases where the two conflict fundamentally.

Closes gaps elsewhere

Ashutosh Singh's Post · g4 — Describes a collaborative eval-dataset creation process — human annotation with reasoning codified into rubrics, inside a continuous cross-functional feedback loop spanning engineering, SMEs, and testing — rather than PMs authoring golden examples alone.

Generative AI in the Real World: Aishwarya Naresh Reganti on Making AI Work in Production

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Closes gaps elsewhere

Related documents