Teaching an AI Agent to Debug Flaky Tests

Igor Kulakov · article · 2026-05-09 source ↗ Reader 🖥️

Agent-Native Design Agent Skills Coding Agents

Flaky tests can be debugged deterministically by diffing code-coverage reports from passing and failing runs to locate where execution paths diverge, and this entire procedure can be packaged as a reusable AI Agent Skill so that an agent follows it without human guidance.

practical · tutorial-narrative — A walkthrough that builds from intuition to tooling to a reusable AI Skill, using a concrete TOCTOU race condition as the running example. Part tutorial, part methodology argument.

Structure

Part	Role
Problem statement: flaky tests undermine CI	Establishes the pain — non-deterministic pass/fail with no code changes wastes human and infrastructure resources
Example project: TOCTOU race in a Spring Boot webshop	Provides a concrete, reproducible flaky test (concurrent checkout producing duplicate invoice numbers)
The intuition: different results imply different code paths	Derives the key insight — if we record and diff the code paths of passing vs. failing runs, the divergence point reveals the root cause
Building the tools: coverage agent + text reporter + diff script	Shows how to (1) run IntelliJ coverage from CLI with hit counting, (2) convert binary reports to text, (3) collect multiple runs and summarize varying lines
Interpreting the diff: hit-count variation pinpoints the race	Demonstrates that the varying lines point exactly to the lazy-init block where the TOCTOU race lives
Packaging as an AI Agent Skill	Converts the manual procedure into a SKILL.md that an agent can discover and execute autonomously
Testing the Skill in a fresh agent session	Validates that a naive prompt ('find and fix the cause of flakiness') triggers skill discovery and correct execution

Key terms

AI Agent Skill Agent Skills
A plain-text document (SKILL.md) in a discoverable location that describes a reusable procedure, conventions, or domain knowledge an AI agent can follow — the AI equivalent of a knowledge base article
flaky test
A test that returns both passes and failures despite no changes to code or test
coverage diff
A comparison of per-line hit counts from coverage reports across multiple test runs, summarizing lines whose execution counts vary between passing and failing runs
hit counting
Recording the number of times each line is executed (not just whether it was hit), enabling detection of concurrency bugs where the same code runs more times than expected
TOCTOU (time-of-check to time-of-use)
A race condition where a thread checks a condition and then acts on it, but another thread changes the state between check and use

Propositions

explicitFlaky tests must take different code paths on passing vs. failing runs; otherwise results would be consistent.conf: high
explicitTest coverage tools, though not designed for tracing, can record the code-path information needed to diagnose flakiness.conf: high
explicitHit counting (execution counts per line) is essential — some flaky bugs differ not in which lines execute but in how many times they execute.conf: high
explicitcorroboratedDiffing coverage reports from passing and failing runs can deterministically point to the root cause of flakiness.conf: medium
supports ←OpenAI Fixes 18-Year-Old GNU libunwind Bug by Treating Crash Debugging Like Epidemiology · p1 — Parallel instance of one principle — comparative/differential data across many cases localizes root cause where single-case inspection fails. There: diffing passing-vs-failing coverage reports. Here: labeling and correlating a year-long population of core dumps to split one syndrome into two clusters.
explicitAI Agent Skills make hard debugging procedures accessible to agents that have no prior context about the problem.conf: high
implicitThe coverage-diff technique generalizes beyond TOCTOU races to other classes of flaky tests.conf: medium
explicitText interfaces are the composable building blocks that allow AI agents to solve complex problems — as long as you can stack text interfaces together, you can teach agents to solve virtually anything.conf: medium

Arguments

Coverage-diff is a sound diagnostic for flaky tests strength: medium

Explicit premises:

P1: Flaky tests must take different code paths
P2: Coverage tools can record code-path information
P3: Hit counting captures frequency differences, not just presence/absence

Implicit premises:

The code-path differences that cause flakiness are observable at the line/branch level of granularity

Inferences:

If flakiness implies path divergence, and coverage captures paths, then coverage-diff reveals the divergence point

Packaging the procedure as an AI Skill makes debugging deterministic strength: medium