Kaggle is making AI benchmark creation effortless
Andrew Wang (Google) · rss · 2026-06-04 Reader 🖥️
By moving Kaggle Benchmarks into developers' local environments and letting AI coding agents author tasks in natural language, Google can scale community-built evaluations to the point where any measurable capability becomes a target that frontier labs will race to improve.
practical · expository — Product announcement framed as a call to action: developers should use Kaggle Benchmarks (now locally, now via AI coding agents) to create evaluation tasks. The expository sections explain the tooling; the closing section argues for why this matters.
Structure
| Part | Role |
|---|---|
| Opening framing: traditional benchmarks are inadequate for reasoning/tool-using agents; the community must build dynamic evaluations. | Problem statement that motivates the rest |
| Recap of Kaggle Benchmarks (10,000+ tasks, public leaderboards) as proof of community uptake. | Evidence of traction |
| Announcement 1: local development support across Antigravity, VSCode, Cursor, coding agents. | Feature claim — removes friction of web-only notebook editor |
| Announcement 2: write-kaggle-benchmarks skill + kaggle-benchmarks SDK + Kaggle CLI let agents author tasks from a natural-language prompt. | Feature claim — removes friction of authoring |
| Closing argument: if a capability can be measured, labs will race to improve it; democratizing eval-creation drives model progress in areas that matter. | Normative conclusion / mission framing |
Key terms
- Benchmark / evaluation task Evals
A discrete, scorable challenge (often a single prompt-and-expected-judgment, e.g. 'is 300+140=460 correct?') submitted to Kaggle and run against models on a public leaderboard. - Community-driven evaluation Evals
Evaluations authored by the broad developer community via Kaggle's tooling, rather than by lab researchers or third-party evaluation organizations. - write-kaggle-benchmarks skill Agent Skills
A Claude/agent skill (instructions + SDK reference) that teaches a coding agent how to translate a natural-language eval description into a runnable Kaggle task. - Trustworthy, transparent
Public leaderboards with reproducible task definitions; the trust is in openness, not in curatorial gatekeeping.
Propositions
- explicitcorroboratedTraditional benchmarks are no longer sufficient for evaluating reasoning agents that write code and use tools.conf: highsupports ←Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation · p2 — Both independently argue that traditional/static benchmarks are no longer adequate once agents are multi-turn, tool-using, and reasoning rather than answering single fixed questions.
- explicitReal-world users of models are better positioned than researchers to author meaningful evaluations.conf: highsupportsPodcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook · p2 — Both argue the most meaningful evals come from real-world users' own domain expertise rather than researcher-built generic benchmarks.
- explicitLowering the authoring cost of benchmarks (local dev + NL agents) will increase the supply of useful evaluations.conf: high
- explicitIf a capability can be measured, frontier labs will race to improve it.conf: high
- implicitcontestedMore benchmarks = better AI progress in the areas that matter.conf: highcontradictsThe AI paradox: More automation, more humans, more work | Dan Shipper · p7 — Kaggle claims more benchmarks drive progress in the areas that matter; Shipper argues benchmarks structurally miss the human framing work that matters most.
- implicitA scorable single-shot prompt (e.g. 'is 300+140=460?') is a legitimate benchmark task.conf: high
- implicitcontestedQuantity and diversity of evaluations outweigh the risk of low-quality, gameable, or contaminated tasks.conf: highcontradictsHow Anthropic's product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code) · p16 — Kaggle's bet that quantity and diversity of evals outweigh quality risks directly opposes Wu's claim that 10 great evals beat hundreds.
- implicitcontestedAI coding agents are competent enough to author rigorous evaluation tasks for AI models — including the model evaluating itself or its peers.conf: highcontradictsGenerative AI in the Real World: Aishwarya Naresh Reganti on Making AI Work in Production · p4 — Kaggle assumes AI agents can rigorously author evals, including self-evaluation; Reganti insists LLM self-evaluation cannot replace manual human effort.
- implicitPublic leaderboards on community-submitted tasks produce signals that labs should optimize toward.conf: high
Arguments
Google should make benchmark authoring effortless and local.
Explicit premises:
- Reasoning agents need richer, dynamic evaluations than the classic suites provide.
- Real-world practitioners know best what matters.
- Until now, authoring meant working in Kaggle's web notebook — friction that suppressed participation.
Implicit premises:
- Volume of community-authored tasks is a good proxy for evaluation quality.
- Coding agents in IDEs are the right authoring substrate (not researchers, not domain experts in non-coding tools).
- Goodhart effects from a 10,000+-task leaderboard ecosystem are manageable or beneficial.
Inferences:
- The center of gravity for eval-creation shifts from research labs and academic consortia to the Kaggle platform — and therefore to Google.
- Whoever owns the eval substrate shapes which capabilities frontier labs optimize for.
Democratizing benchmark creation benefits humanity. strength: low
Explicit premises:
- Evaluations must reflect the full diversity of real-world challenges.
- If a capability can be measured, labs will improve it.
Implicit premises:
- Measured capabilities map cleanly to capabilities humanity wants improved.
- Lab optimization is the bottleneck on AI benefiting humanity (not deployment, alignment, governance, distribution).
- Goodhart's law — that measures become targets and cease to be good measures — does not apply at scale here.
Inferences:
- There is no acknowledged downside to a flood of new benchmarks; the failure mode is not enumerated.
Narratives
- The example task: an agent asked to build a benchmark that tests whether the model can judge '300+140=460 is correct?' — a featured math-false-statement task on Kaggle.
→ Shows what 'effortless' means in practice — and implicitly defines the floor of what counts as a benchmark task. Suggests trivially-scorable, single-shot judgments are the unit of measurement. - The '10,000+ tasks since launch' figure as a success metric.
→ Implicitly argues that volume is the right success metric for an eval marketplace — and that growth in task count is itself progress.
Questions addressed
- ✅ How do we evaluate reasoning agents when traditional benchmarks are saturated or inadequate?
Let the community author dynamic, real-world-grounded tasks via Kaggle, now from local IDEs with agent assistance. - ✅ Who is qualified to author a benchmark task?
Anyone with an IDE and a coding agent. - ⭕ How do we prevent gaming, contamination, or Goodhart drift in a leaderboard of 10,000+ community-authored tasks?
- ⭕ What is the quality-control mechanism for tasks submitted by AI agents writing tasks for AI agents?
- ⭕ Does running on a Google-owned eval platform create a conflict of interest when Gemini models are on the same leaderboards?
Gaps
- incompleteNo discussion of Goodhart's law — that turning a measure into a target erodes its value as a measure. With 10,000+ vendor-friendly tasks and labs explicitly racing to improve scores, this is the central risk the piece sidesteps.
Read next: Manheim & Garrabrant on Goodhart taxonomies; recent evaluation-contamination work (Sainz et al., Zhou et al.); Gary Marcus on benchmark-overfitting in LLM math performance. - incompleteNo quality-control story: who reviews tasks, who detects duplicates or trivially-gameable items, who flags contamination when the same agents author both tasks and solutions.
Read next: HELM and BIG-bench review processes; METR/Apollo on adversarial eval design; the DORA 2025 finding that AI productivity gains correlate with deteriorating evaluation rigor. - incompleteConflict-of-interest framing is absent. Kaggle is Google; Gemini is Google; leaderboards influence purchasing and research narrative. A genuinely 'trustworthy' eval substrate needs a story for vendor neutrality.
Read next: MLCommons governance model; Stanford HELM independence claims; AI Snake Oil on platform-vendor evaluation. - incompleteThe implicit definition of 'benchmark' collapses to 'any scorable single-shot prompt.' This may be category-confusion with what the eval community means by 'benchmark' (a curated, statistically powered suite).
Read next: Liang et al. on HELM construct validity; recent agentic-eval work (SWE-bench Verified, GAIA) on what makes a task suite vs. a task-bag. - uninformedNo engagement with the mounting literature showing LLM math benchmarks are heavily contaminated and that pass-rates do not transfer — the very example task ('300+140=460') is in the genre most criticized.
Read next: Marcus & Davis on LLM arithmetic; Apple's GSM-Symbolic paper; arithmetic reasoning critiques 2024-2026.
Related documents
- Podcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook
- The AI paradox: More automation, more humans, more work | Dan Shipper
- Generative AI in the Real World: Aishwarya Naresh Reganti on Making AI Work in Production
- How Anthropic's product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code)
- Agent Skills