Podcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook

Sam Bhagwat · rss · 2026-04-10 source ↗ Reader 🖥️

Agent Memory Agentic Loop Career Development Enterprise AI Adoption & Readiness Evals LLM Limitations Non-Deterministic Software Vibe Coding

Shipping agentic AI applications to production requires organizations to combine software engineering rigor with data science's comfort with statistical uncertainty, using cross-functional Tiger Teams, domain-specific evals built on proprietary data, and a culture of enthusiastic experimentation.

practical · interview / podcast transcript — A practitioner interview covering open source community management, AI engineering practices, eval strategies, and team structures for agentic applications. Primarily prescriptive rather than descriptive.

Structure

Part	Role
Introduction and open source background	Establishes speaker credibility and contextualizes AI engineering within the broader open source ecosystem
Open source community health	Demonstrates the tension between opinionated beginnings and flexible maturity
Balancing open source spirit with commercial reality	Identifies the personnel archetype needed — neither purist nor pure profit-seeker
AI engineering fundamentals: agents, workflows, evals	Core technical content defining the building blocks
Writing effective evals for agentic applications	Central practical argument — domain-specific evals are the highest-value activity
Marrying software engineering and data science mindsets	Identifies the cultural gap that Tiger Teams solve
Tiger Teams and team archetypes	Prescriptive organizational guidance
Closing advice: lean into discomfort	Motivational framing for individual practitioners

Key terms

evals Evals
Statistical tests applied to agentic application outputs to measure accuracy, fairness, and domain-specific correctness — analogous to but 10x more important than traditional software tests
Tiger Team
A cross-functional team assembled specifically for an AI project, pulling people from engineering, data science, and product roles outside the normal org chart
agentic applications Agentic Loop
Software where an LLM runs in a loop, calling tools and maintaining memory, deployed as a production interface (e.g., a client within a SaaS application)
vibe coding Vibe Coding
Using AI coding assistants (Claude Code, Cursor) for a co-creative development experience — distinct from building agentic applications
memory Agent Memory
A structured compression of a queue of messages — working memory, semantic memory, observational memory
non-determinism Non-Deterministic Software
The property of agentic applications where multiple successful runs can produce different response bodies, unlike traditional software

Propositions

explicitAI engineering is following a similar adoption pattern to DevOps and data engineering but at roughly three to four times the speed.conf: medium
explicitestablishedThe highest-value evals are those written against an organization's own unique data and domain expertise, not generic off-the-shelf benchmarks.conf: high
supports ←A pragmatic guide to LLM evals for devs · p2 — Both make the same claim from different angles: the highest-value evals are built from an organization's own data/domain expertise, not generic off-the-shelf benchmarks/metrics.
supports ←Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation · p6 — Both converge on the claim that the highest-value evaluation scenarios are built from an organization's own domain knowledge and tool capabilities, not generic off-the-shelf tasks.
supports ←Downstream #1: Where Do Our Tokens Go? · p7 — Convergent claim that the highest-value evals are written against an organization's own data/workflows, not generic benchmarks; Uber's shadow-comparison methodology is a concrete instance.
supports ←Kaggle is making AI benchmark creation effortless · p2 — Both argue the most meaningful evals come from real-world users' own domain expertise rather than researcher-built generic benchmarks.
explicitestablishedEvals are 10x more important in AI engineering than tests are in traditional software engineering.conf: medium
supports ←A pragmatic guide to LLM evals for devs · p4 — Both position evals as a distinct, higher-stakes discipline than traditional software testing for AI systems — this article supplies the concrete binary-PASS/FAIL mechanism for why that discipline looks different.
supports ←How Anthropic's product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code) · p16 — Wu's few-great-evals-underappreciated claim corroborates Bhagwat's claim that evals carry outsized importance in AI engineering.
supports ←Presentation: Engineering AI for Creativity and Curiosity on Mobile · p12 — Both treat rigorous, explicit evaluation as the load-bearing discipline for shipping AI features safely — this talk supplies a concrete mechanism ('engineer the metric' + human-in-the-loop review) for the abstract claim that evals matter far more in AI engineering than tests do in traditional software.
explicitShipping AI agents to production requires marrying software engineering rigor with data science's comfort with statistical uncertainty.conf: high
explicitCross-functional Tiger Teams are necessary because agentic AI projects do not map to existing org structures.conf: high
explicitOpen source communities must evolve from opinionated beginnings to flexible maturity, decoupling maintainer identity from specific technical approaches.conf: high
explicitThe current moment is a career window: because few people have deep AI engineering experience, motivated practitioners can break in more easily than in mature fields.conf: medium
explicitVibe coding (using AI coding assistants) and building agentic applications are two distinct activities that should not be conflated.conf: high
implicitOrganizations that are more command-and-control struggle to ship agentic applications because they cannot easily form cross-functional Tiger Teams.conf: high
implicitcorroboratedThe competitive advantage in AI-powered applications will come from proprietary data and domain expertise, not from model capabilities alone.conf: high
supports ←Everyone Is Talking About Context for AI. Here's What Most Companies Still Miss. · p3 — Both argue that proprietary data/domain specificity, not raw model capability, is what makes AI valuable/defensible in an enterprise setting.
explicitFeature-flagged staged rollouts are the appropriate deployment strategy for agentic applications, with timescales of weeks rather than days.conf: high
implicitDefault skepticism toward new technology is a greater career risk for experienced engineers than default enthusiasm.conf: medium

Arguments

Organizations should invest primarily in domain-specific evals rather than generic benchmarks. strength: high

Explicit premises:

Model providers already train and eval against public benchmarks and datasets.
The things important for your application are unique to your organization's area of core competence.
The data your organization has is not publicly present in model training data.

Implicit premises:

Generic evals provide diminishing returns because model providers are already optimizing for them.
The gap between model capability and production readiness is filled by domain-specific quality assurance.

Inferences:

Therefore the highest marginal value of eval effort is in proprietary, domain-specific test suites.

Cross-functional Tiger Teams are the right organizational structure for shipping agentic applications. strength: high

Explicit premises:

Data scientists are comfortable with statistical uncertainty but not production software.
Software engineers build production systems but aren't trained in statistical thinking.
Agentic applications require both skillsets simultaneously.

Implicit premises:

Traditional org structures separate these disciplines into different teams or departments.
A single person rarely possesses both deep software engineering and data science skills.
The speed of AI adoption (3-4x faster than previous waves) means organizations cannot wait for organic skill diffusion.

Inferences:

Therefore a cross-functional team structure that pulls from both disciplines is necessary.

Experienced engineers should overcome default skepticism and embrace AI engineering. strength: medium

Explicit premises:

As engineers age, they shift from default enthusiasm to default skepticism.
Being good in a new field requires being okay with discomfort and being bad at something initially.
There is significant career opportunity in being an early adopter.

Implicit premises:

AI engineering is not a passing fad — it is comparable to DevOps and data engineering in staying power.
The cost of missing this wave is higher than the cost of temporary discomfort.

Inferences:

Therefore the rational strategy is to lean into the discomfort of learning AI engineering.

Narratives

Gatsby's evolution from a React framework built to scratch an itch by two friends to a popular open source project requiring community management and commercial pragmatism.
→ Establishes credibility and provides a concrete example of the open source community lifecycle pattern.
An HR SaaS platform observed users exporting CSVs and pasting them into ChatGPT, leading the company to build an in-app agent that could answer HR questions using organizational data.
→ Concrete example of the modal use case for agentic applications, illustrating both the privacy problem and the domain knowledge advantage.
Mastra's 'Kindergarten' Slack channel where the team shares links and pairs frequently to diffuse AI tool expertise.
→ Illustrates the humility and collaborative culture needed for AI-augmented engineering teams.
A CTO of a 150-200 person engineering org personally writing code and leading the agentic application project.
→ Illustrates that AI projects are high-value enough to warrant senior leadership directly contributing, and that cross-functional leadership is critical.

Questions addressed

✅ How should organizations structure teams to ship agentic AI applications?
Use cross-functional Tiger Teams that combine software engineering and data science skills, outside normal org structure.
✅ What makes evals effective for agentic applications?
Evals built on organization-specific data and domain expertise, created with subject matter experts, measuring accuracy against comprehensive domain-specific question sets.
✅ How should agentic applications be deployed to production?
Staged rollout using feature flags: beta testers, then 1%, 5%, 10%, 50%, over weeks not days.
✅ How fast is AI engineering being adopted compared to previous technology waves?
Roughly 3-4x faster than DevOps and data engineering adoption.
⭕ What specific accuracy thresholds should organizations target for agentic applications?
Acknowledged as context-dependent ('sometimes higher, sometimes lower') but no framework for determining thresholds is offered.
⭕ How do you maintain quality and reliability of AI coding agents used in the development process itself?
Not addressed — the discussion of evals focuses on product agents, not development workflow agents.

Gaps

incompleteThe '3-4x faster' adoption claim is stated without supporting data, methodology, or definition of what metric is being measured. What specific growth metrics are being compared?
Read next: Look for empirical studies comparing technology adoption curves (e.g., Thoughtworks Technology Radar data, Stack Overflow surveys, or job posting analysis).
incompleteNo discussion of how to determine appropriate accuracy thresholds for different risk levels. The framework jumps from 'build evals' to 'hit your threshold' without addressing how to set the threshold.
Read next: Risk management frameworks for AI systems, such as NIST AI Risk Management Framework or EU AI Act risk categories.
uninformedNo mention of adversarial testing, security evals, or prompt injection risks for customer-facing agentic applications. The HR SaaS example involves sensitive data but security is only mentioned as a motivation (CSV-to-ChatGPT privacy concern), not as an eval category.
Read next: OWASP Top 10 for LLM Applications; adversarial ML literature; prompt injection defense strategies.
closed byHacking Meta's AI Chatbot — A concrete adversarial case for customer-facing agents: a production chatbot with tool access to account state was social-engineered into account takeover — exactly the security-eval risk category the gap flags as absent.
incompleteThe Tiger Team concept is described but there is no discussion of how Tiger Teams dissolve back into the organization, how knowledge transfers, or how the org evolves its permanent structure based on what Tiger Teams learn.
Read next: Literature on temporary organizations, project-to-product transitions, and Team Topologies by Skelton and Pais.
misinformedThe distinction between vibe coding and building agentic applications is drawn too sharply. In practice, development workflow agents (the Discord-to-GitHub-issue agent described) are themselves agentic applications that need evals, blurring the boundary Bhagwat draws.
Read next: Inner loop / outer loop development frameworks; the concept of 'dogfooding' applied to AI agent development.

Podcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Related documents