Research-Driven Agents: What Happens When Your Agent Reads Before It Codes
Alex Kim · article · 2026-04-09 source ↗ Reader 🖥️
Coding agents generate better optimizations when they read papers and study competing projects before touching code, because a codebase tells you what the code does but not why it's slow or what alternatives exist outside the repo — and the quality of an agent's hypotheses is bounded by the quality of its context.
practical · mixed — Practical: argues for a specific architectural change to coding agent loops (add a research phase). Mixed form: part technical case study with concrete benchmarks, part generalizable methodology recommendation. The evidence is unusually specific — actual code diffs, benchmark numbers, cost breakdowns — making it one of the more rigorous agent-capability posts available.
Structure
| Part | Role |
|---|---|
| Where code-only context works (Liquid template engine case) | Establishes the baseline — shows that code-only agents CAN work when the optimization surface is visible in the source. This prevents the argument from overstating the research phase's necessity. |
| Where code-only context breaks down (llama.cpp Wave 1 failures) | Primary evidence for the problem — the agent tried SIMD micro-optimizations and got ~0.8% gains at best, because it couldn't know from code alone that the bottleneck was memory bandwidth, not compute |
| Adding a research phase (papers, forks, competing backends) | The intervention — describes the architectural change to the agent loop |
| What the research turned up (ik_llama.cpp, Blockbuster paper, CUDA backend analysis) | Evidence that external context produces qualitatively different hypotheses — the agent started asking 'can I fuse operations?' instead of 'can I make this loop faster?' |
| The five optimizations that landed (softmax fusion, RMS norm fusion, adaptive parallelization, graph-level fusion, flash attention KQ fusion) | Concrete evidence — each optimization traced to a specific insight from the research phase, with before/after code and benchmark numbers |
| What didn't work (25 failed experiments, benchmark bug, noisy VMs) | Credibility and methodology — honest about the failure rate (5/30+), the debugging challenges, and the measurement problems. This section is unusually candid for a product blog. |
| Try it on your own project (setup instructions, candidate projects) | Call to action — positions this as a reusable methodology, not a one-off result |
Key terms
- hypothesis quality Taste & Judgment
The quality of the optimization ideas an agent generates before running experiments. The core claim is that hypothesis quality — not execution capability — is the binding constraint on agent performance. A code-only agent generates shallow hypotheses (micro-optimizations); a research-informed agent generates structural hypotheses (operator fusions). - research phase
A step added before the experiment loop where the agent reads arxiv papers, studies competing implementations, and analyzes how other backends solve the same problem. Analogous to what a senior engineer would do before touching unfamiliar code. - code-only context
The information available to an agent when its only input is the source code of the target project. Sufficient when the optimization surface is visible in the source (Liquid), insufficient when the answer lives in domain knowledge, papers, or competing implementations (llama.cpp). - operator fusion
Combining multiple sequential operations (e.g., copy + scale + mask) into a single pass over the data, eliminating intermediate memory writes. The primary optimization pattern discovered through the research phase. - autoresearch loop Recursive Self-Improvement
The iterative cycle of: brainstorm idea → edit code → run experiment → check metric → keep or discard. Originally from Karpathy's autoresearch project. The authors' contribution is adding a research step before brainstorming.
Propositions
- explicitThe quality of an agent's hypotheses is bounded by the quality of its context — code alone is insufficient for problems where the answer lives outside the codebaseconf: high
- explicitStudying competing forks and other backends was more productive than searching arxiv papersconf: high
- explicitAdding a research phase to a coding agent loop changes the TYPE of hypothesis generated, not just the quality — from 'make this loop faster' to 'can I fuse these operations?'conf: high
- explicitThe compiler and hardware are already doing many of the micro-optimizations a naive agent would try — without experience with compiler behavior, the agent wastes experimentsconf: high
- explicitA research-informed agent working on 4 VMs for 3 hours at $29 total cost can find meaningful optimizations in a mature C++ project (llama.cpp)conf: high
- implicitThe experiment success rate is low (5/30+, ~16%) and this is acceptable — the cost per experiment is low enough that a high failure rate is fineconf: high
- implicitWhat makes a senior engineer valuable is not coding speed but the ability to generate good hypotheses from broad context — and this is exactly what the research phase replicatesconf: high
- explicitAgents trust their own instrumentation too much — the benchmark bug went undetected across multiple experiment waves because the agent didn't question unreasonable numbersconf: high
- explicitWell-optimized mature codebases (Redis, PostgreSQL, CPython, SQLite) are harder targets — decades of human optimization leaves less surface for agents to findconf: high
Arguments
Coding agents should read before they code — a research phase that studies papers, forks, and competing implementations produces qualitatively better hypotheses strength: high
Explicit premises:
- Wave 1 (code-only) produced only micro-optimizations within noise (~0.8%)
- After the research phase, the agent discovered operator fusion opportunities invisible from code alone
- 5 optimizations landed, producing +15% TG improvement on x86
- The fork analysis (ik_llama.cpp, CUDA backend) directly informed 2 of 5 optimizations
Implicit premises:
- The autoresearch loop's bottleneck is hypothesis generation, not experiment execution — adding more VMs doesn't help if the hypotheses are shallow
- Domain knowledge is transferable across implementations — what CUDA does is relevant to CPU optimization because the mathematical operations are the same
- The research phase is worth the time cost even though it reduces experiment throughput (30 experiments in 3 hours vs. 910 in 8 hours for GPU autoresearch)
Inferences:
- This generalizes beyond optimization: any agent task where the answer lives outside the immediate codebase would benefit from a research phase — bug diagnosis, architecture decisions, API design
- The research phase is essentially giving the agent what a senior engineer already has: broad context from years of reading papers, studying other systems, and understanding the domain. The agent needs to acquire this context explicitly; the senior has it implicitly
- This is the same insight as Kvam's 'spec first, then code' — front-loading context produces better output. The research phase is spec-writing for agents
Agents have a systematic weakness in calibrating confidence against their own instrumentation strength: medium
Explicit premises:
- The benchmark bug reported 14 t/s instead of 52 t/s and went undetected across multiple waves
- A human would probably spot unreasonably low numbers sooner
- The agent trusted its own script
Implicit premises:
- Agents lack the common-sense sanity checking that humans apply to measurement outputs — they don't have a prior expectation of what 'reasonable' looks like
- Self-generated instrumentation is particularly dangerous because the agent has no external reference point
Inferences:
- Agent loops should include a sanity-checking step that compares measurements against known baselines or expected ranges — a form of metacognition
- This is a general agent failure mode: agents are bad at questioning their own observations. It applies beyond benchmarking to any agent that generates and then acts on its own data
Narratives
- Wave 1 failure: the agent tried SIMD micro-optimizations on llama.cpp's dot products and got nowhere, then diagnosed itself — 'text generation is memory-bandwidth bound, not compute bound'
→ Demonstrates the code-only hypothesis trap in action. The agent did something reasonable but wrong, because the information needed (hardware memory bandwidth, roofline model) wasn't in the code. Also shows agents can self-diagnose — the pivot was the agent's own conclusion. - Shopify's Liquid optimization: 93 commits, 53% parse+render improvement, zero regressions — done by a code-only agent
→ Establishes the positive case: code-only works when the optimization surface is visible. Prevents the argument from being read as 'agents always need research.' The Liquid tokenizer's bottleneck was visible in the source. - The RMS_NORM + MUL fusion discovery: the agent noticed CUDA and Metal backends fuse these operations but the CPU backend doesn't — an insight impossible from CPU code alone
→ The clearest single example of research-driven discovery. The optimization exists in every other backend and the agent would never have looked for it without cross-backend analysis. This is the proof case for the research phase. - The benchmark bug: JSON parsing error caused wrong baselines for multiple experiment waves
→ Demonstrates agent overconfidence in self-generated instrumentation. Also serves as an honesty signal — the authors report their own tooling failures.
Questions addressed
- ✅ Can coding agents optimize mature C++ codebases?
Yes, with a research phase. Code-only agents get shallow results (~0.8%), but research-informed agents found meaningful optimizations (+15% TG) in llama.cpp. - ✅ What determines the quality of agent-generated optimizations?
Hypothesis quality, which is bounded by context quality. Broader context (papers, forks, other backends) produces structurally different and better hypotheses. - ✅ Is the research phase worth the time cost given lower experiment throughput?
Yes — 5/30 experiments at $29 in 3 hours produced better results than 910 experiments in 8 hours at $300 on a different problem. Quality over quantity. - ⭕ Does this generalize beyond ML inference optimization?
- ⭕ How should an agent decide WHEN it needs a research phase vs. when code-only context is sufficient?
Gaps
- incompleteThe article doesn't address how the agent decides whether it needs a research phase. The Liquid case didn't need one; the llama.cpp case did. But the agent couldn't have known this in advance. A decision heuristic ('when should I read before coding?') is the missing meta-skill that would make this approach self-directing rather than human-configured.
Read next: Metacognitive agent architectures; 'knowing what you don't know' in AI systems; confidence calibration research - incompleteThe claim that well-optimized codebases (Redis, PostgreSQL, SQLite) are harder targets is mentioned in one sentence but not explored. This is actually the most interesting boundary: at what point has a codebase been optimized enough that agent-driven improvement becomes uneconomical? This would define the useful scope of the entire approach.
Read next: Performance engineering literature on diminishing returns; the 'last mile' problem in optimization - incompleteThe research phase is described as reading papers and studying forks, but the article doesn't discuss how the agent evaluates which papers and forks are relevant. With thousands of arxiv papers and potentially dozens of forks, the agent needs a research strategy — and that strategy itself requires domain knowledge.
Read next: Information retrieval for technical literature; RAG architectures for agent research; the 'research taste' problem - incompleteThe article is honest about agent overconfidence (the benchmark bug) but doesn't propose a solution. How should agents sanity-check their own instrumentation? This is a general agent reliability problem that extends far beyond optimization benchmarks.
Read next: Agent self-verification; Anthropic's work on AI oversight; the 'quis custodiet' problem in autonomous systems
Related documents
- Jacob Langvad Nilsson's Post
- Don't train the model, evolve the harness
- 20 Loop Design Patterns Every AI Engineer Should Know
- Agent Skills: Google-Backed AI Coding Agent Workflow
- Agentic AI and Code Reviews
- Accelerating Delivery: How AI Agents Can Own the Initial Code Draft
- [FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero
- AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering
- The Almedra Token-Efficiency Benchmark, Updated For GLM 5.2
- The Anatomy of an Agent Harness