How Datadog Used Claude and Cursor for Test-Driven Production Migration
Sergio De Simone · rss · 2026-07-10 Reader 🖥️
Coding AgentsFeedback LoopsLLM LimitationsThe Oracle (Referenced Evaluation)Test-Driven DevelopmentVerifiability (as the Quality Bar)
Datadog's AI-assisted migration of a critical production system off a key-value backend onto a relational one succeeded not because Claude and Cursor generated code autonomously, but because a strong human-authored test suite plus a live blue/green validator supplied the pass/fail oracle that bounded and verified every AI-generated change — making the coverage of that verification, not the capability of the model, the ceiling on how much AI output could be trusted.
practical · expository practitioner news report — A short InfoQ secondary summary of a first-party Datadog engineering blog post (a case-study retrospective). It reports on how-to knowledge — how to run an AI-assisted, test-driven production migration — so it is practical: it aims at directing action, not establishing a theory.
Structure
| Part | Role |
|---|---|
| The problem | sets up the need for migration |
| The schema redesign | establishes the target the AI refactor aims at |
| The AI-assisted refactoring method | core practice described |
| The three enabling elements | explains why it worked |
| The three phases | sequences the execution |
| Where the approach fell short | qualifies the success |
| The results and conclusion | delivers the verdict |
Key terms
- Test-driven refactoring / migration
A per-method migration loop in which a human-authored failing test defines the target behavior, the AI generates an implementation against the old code + new schema, and the passing/failing test — not the human reading the code — decides whether the output is accepted. - Pass/fail criterion (the test suite as oracle)
A comprehensive test suite treated as the clear, mechanical accept/reject gate for every AI-generated change — the artifact that converts non-trusted AI output into trusted output. - Blue/green with a validator service
Running the old FoundationDB-backed and new PostgreSQL-backed Stream Routers side by side on the same live requests, with a dedicated validator per cluster continuously comparing their routing responses and alerting the instant they diverge — a correctness check referenced live from the running old system rather than from frozen test cases. - Code modularity
The property that let the new Stream Router implement the same API on PostgreSQL instead of FoundationDB without requiring changes elsewhere — the structural precondition that made a swap-the-backend migration tractable at all. - Suboptimal queries / niche optimizations
Queries that return the correct results but issue far more round trips than necessary; the specific optimization patterns (batching, UNNEST tricks, common table expressions) that the AI could not discover on its own even though they exist in the literature, and had to be supplied by humans before the model could replicate them.
Propositions
- explicitThe strength of Datadog's test suite was the key success factor of the migration, because it ultimately determined how much they could trust AI-generated code.conf: highsupportsDoorDash Used Copilot to Convert Its XCTest-Based iOS Test Suite to Swift Testing · p2 — Datadog's conclusion that the test suite's strength set the trust ceiling is a second production instance of DoorDash's claim that AI can reliably automate mechanical code migration when paired with strict reliability gates — here the gate is the comprehensive test suite plus the live validator.
- explicitcontestedThe AI was not generating code autonomously: for each method the humans supplied the old implementation, the new schema, and a failing test, and the tests told them whether the model's first pass was correct.conf: highsupportsSlack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation · p6 — In both cases the pass/fail oracle stays human-authored: Slack's agent runs are validated against engineer-defined assertions; Datadog's per-method AI output is judged by human-written failing tests. Different mechanism (agent executes tests vs agent writes code) but the same load-bearing principle — humans own the acceptance oracle.supportsFive studies that are changing how I think about AI in software engineering · p5 — The non-autonomous loop — humans supplying schema, tests, and validation while the model produces first-pass code — is concrete field evidence for AI and human work being complements, not substitutes, because substantial human work remains to specify, verify, and integrate.contradicts ←Practices I Abandoned with Agents: An Ode to Test-Driven Development · p3 — Datadog's migration succeeded by keeping agents on a tight per-method leash with a failing test for each small unit — the granular, test-first discipline Tornhill specifically abandoned as unproductive for agents. Both keep 'test first,' but disagree sharply on granularity.
- explicitThree elements made the migration possible: strong code modularity (same API on a different backend), a comprehensive test suite serving as a clear pass/fail criterion for every AI change, and a parallel dual-instance infrastructure routed via feature flags.conf: high
- explicitA dedicated validator service in every cluster continuously compared the routing responses of the old and new Stream Routers on the same live requests and alerted immediately on any divergence.conf: highsupportsWhat Survived Contact · p13 — Datadog's blue/green validator is a textbook real-world instance of the referenced-live acceptance oracle: expected answers are referenced fresh from the running old system per request rather than authored and frozen, so it covers every live input rather than a handful of enumerated scenarios.
- explicitClaude produced correct but suboptimal queries and did not independently discover known optimization patterns (batching, UNNEST tricks, common table expressions) even though they exist in the literature; humans had to write the optimized versions first, after which the models could replicate the pattern in later methods.conf: highsupportsFive studies that are changing how I think about AI in software engineering · p1 — The AI compressed the upstream code-generation (translating each method to the new schema) while the binding work moved downstream to humans who had to review the output and author the query optimizations the model could not discover — an instance of the constraint shifting downstream to verification and refinement.
- explicitThe migration produced large production gains: operations fell from ~45 minutes to about a second, latencies from hundreds of milliseconds to a few, data storage became up to 40x smaller, and database costs were reduced by roughly 90%.conf: high
- implicitAI-assisted migration works by decomposing the task into per-method units each bounded by a concrete, checkable specification (old code + new schema + failing test) rather than by a high-level goal, because higher-level prompts proved less effective.conf: medium
- implicitThe test suite functions as an external verification oracle that substitutes for the human's ability to read and trust each individual AI change, so the reliability ceiling of AI-assisted refactoring is set by the coverage and quality of that oracle.conf: high
- implicitAI accelerates the mechanical translation layer of a migration but not the creative optimization layer, so human expertise shifts to authoring the schema, the tests, and the non-obvious performance patterns.conf: high
Arguments
The migration succeeded because of the verification infrastructure (test suite + live validator), not because the AI could autonomously produce trustworthy code. strength: high
Explicit premises:
- The tests were the pass/fail criterion that told the team whether each AI-generated method was correct.
- The models were not generating code autonomously — humans supplied the old implementation, new schema, and a failing test per method.
- A live validator continuously compared old and new routers on real requests and alerted on divergence.
- Wakim explicitly concludes the test suite's strength determined how much AI-generated code they could trust.
Implicit premises:
- AI-generated code cannot be trusted on its own and requires an external, mechanical check before acceptance.
- A sufficiently comprehensive, checkable oracle can substitute for a human reading and vouching for each change.
- How much of the AI's output you can safely accept is bounded by how much of the intended behavior your verification actually covers.
Inferences:
- Therefore the binding constraint on AI-assisted migration is the quality of the verification harness, not the capability of the model.
- Therefore the engineering effort that pays off is invested in tests, schema design, and comparison tooling rather than in more autonomous prompting.
AI accelerates mechanical code migration but high-level optimization judgment remains a human responsibility. strength: high
Explicit premises:
- AI-generated queries returned correct results but issued far more round trips than necessary.
- The models did not discover known optimization patterns (batching, UNNEST, CTEs) independently, though these exist in the literature.
- Humans wrote the optimized versions, after which the models could replicate the pattern in subsequent methods.
- Higher-level prompts had limited effectiveness.
Implicit premises:
- LLMs apply patterns present in their immediate context rather than reliably retrieving and instantiating techniques merely because they exist 'in the literature'.
- Replication-from-an-exemplar is a different and easier capability than discovery-from-first-principles.
Inferences:
- Therefore providing a worked exemplar, not a higher-level instruction, is the effective way to get optimized output from the model.
- Therefore the non-obvious performance work stays with humans even in an otherwise AI-driven migration.
Narratives
- The Datadog Stream Router migration told as 'what worked, what didn't': a KV-backed API that reconstructed relational logic in-process and took up to 45 minutes on its hardest operations was refactored, method by method, onto a relational PostgreSQL schema using Claude and Cursor against failing tests, deployed blue/green against the old system, and ended with ~1s operations, 40x smaller storage, and ~90% lower DB cost.
→ Serves as the single concrete existence proof grounding the article's thesis that a strong test suite (and a live comparison oracle), not model autonomy, is the enabling condition for trustworthy AI-assisted migration of a critical production system.
Questions addressed
- ✅ What made an AI-assisted, test-driven migration of a critical production system actually succeed?
A comprehensive human-authored test suite acting as the pass/fail oracle per method, strong modularity that let the same API run on a new backend, a per-method loop feeding the AI old code + new schema + a failing test, and a live blue/green validator comparing old and new responses on real traffic. - ✅ Where did the AI-assisted approach fall short?
Higher-level prompts were less effective than narrow test-anchored ones; Claude generated correct-but-suboptimal queries and could not discover known optimizations independently; and token consumption was high due to feeding full test-output dumps and the iterative test/code/schema loop. - ✅ Can AI be trusted to refactor a critical production system?
Conditionally yes — trust is proportional to the strength of the verification harness: the test suite gates each change and the live validator catches any divergence from the old system in production.
Gaps
- incompleteThe blue/green validator verifies behavioral EQUIVALENCE to the old Stream Router, not correctness: it checks that the new router's responses match the old one's on each live request. Any latent bug or quirk in the old FoundationDB-backed system would be faithfully preserved and pass the validator silently, and the per-method TDD loop likewise inherits whatever blind spots the human-authored failing tests contain. The article never examines this — it treats 'agrees with the old system / passes the tests' as equivalent to 'correct', so the oracle's own fidelity to true intent is unquestioned.
Read next: Read against extract-what-survived-contact#g4 (the Goodhart case where a passable oracle diverges from true, uncheckable intent) and the reference-oracle concept: a referenced-live oracle covers every input but only relative to the reference's own behavior. - incompleteThe human/AI division of labor is characterized qualitatively (AI does first-pass translation; humans supply schema, tests, and optimizations) but never quantified: no data on what fraction of methods the AI got right on the first pass versus needed manual fixes, how many iterations the loop took, or how the AI-driven migration compared in effort or time against a conventional hand-refactor.
Read next: Read the original Datadog engineering post (datadoghq.com/blog/engineering/ai-assisted-storage-migration) for any first-party figures the InfoQ summary dropped; then compare with extract-doordash-copilot-ios-test-migration#g1, which flags the same missing intervention-fraction data for an AI-assisted test migration. - incompleteHigh token consumption is named as a shortcoming but never costed: there is no total token or dollar figure for the migration, so the economic viability of the AI-assisted approach relative to a manual refactor cannot be assessed — the impressive 90% DB-cost and 40x storage wins are outcome metrics of the new architecture, not of the AI method itself.
Read next: Read the original Datadog post for token/cost budgets; read alongside ai-unit-economics and token-cost analyses to weigh the method's cost against its speed.
Related documents
- Five studies that are changing how I think about AI in software engineering
- What Survived Contact
- Slack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation
- [FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero
- Have your agent record video demos of its work with shot-scraper video
- Agentic AI and Code Reviews
- AI Demands More Engineering Discipline, Not Less
- Google's AlphaEvolve Reaches General Availability with Evolutionary Code Optimization as a Service
- Architecture as Code to Teach Humans and Agents About Architecture
- AWS Expands DevOps Agent with AI-Powered Release Management to Validate Code Before Production