AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering

Mark Silvester · rss · 2026-07-25 Reader 🖥️

Agent Harness Agentic Loop Agentic vs. Deterministic System Design Context Engineering Evals Frontier Model Convergence / Commoditization Open-Weight Models SRE & Incident Response

The hard part of AI-assisted root cause analysis has shifted from model reasoning -- which recent evidence suggests is largely solved by frontier and even many open-weight models given the same prepared context -- to context engineering: the harness work of correlating signals and assembling a compact, unmisleading context before the model is ever called.

practical · mixed — Primarily an expository news report on Coroot's Nikolay Sivko's research, but the article is structured argumentatively around a single claim (harness > model for AI RCA) and closes with an explicit practical recommendation ('effort spent preparing context may pay off more than reaching for a larger model'), so it functions as a practical text wearing expository clothing.

Structure

Part	Role
Opening framing: two camps in AI RCA design (agent-based vs. deterministic) and the diagnostic problem that failures in either can't be cleanly attributed to weak reasoning or bad harness	context
Coroot/Sivko's research: splitting RCA into 'reasoning over the data' vs. 'the harness that decides what data reaches the model', framed as two separately measurable jobs	evidence setup
The experiment: one Chaos Mesh NetworkChaos scenario with deliberately misleading signals, run as a single ~9,800-token prompt against eleven models	evidence
Results: closed frontier models and most larger open-weight models pass; smaller open-weight models split (one passes, two fail)	evidence
Counterweight: agent-based designs' genuine advantage for novel incidents, and the operability problems of multi-agent investigation in production (ZenML/Incident.io, Reddit practitioner reports)	complicating evidence / concession
Cost note and closing synthesis: reasoning 'basically solved', the real work is 'preparing the right, compact context', tied to the wider context-engineering discourse (Anthropic, LangChain, Mezmo)	conclusion

Key terms

context engineering Context Engineering
The discipline of curating the smallest set of high-signal, compact context supplied to a model before it reasons -- treated as the emerging hard problem in AI RCA and, per the article's closing citations (Anthropic, LangChain, Mezmo), in AI-agent engineering generally.
the harness
The pipeline/logic that decides what data reaches the model and in what shape -- explicitly separated from the model's own reasoning as the second of RCA's 'two jobs' (Sivko's framing, endorsed by the author).
deterministic (RCA) design Agentic vs. Deterministic System Design
An architecture that correlates signals into findings up front and hands the model one fixed, prepared context with no agent loop -- as instantiated by Coroot and (per the article) Dynatrace's Davis AI.
agent-based (RCA) design Agentic vs. Deterministic System Design
An architecture that hands the model tools and lets it decide what telemetry to fetch as it reasons, i.e. an open-ended investigation loop.

Propositions

explicitThe hard part of AI-assisted root cause analysis is no longer the model's reasoning ability, but the harness/pipeline that decides what data reaches the model and in what shape.conf: high
supportsChatGPT Just Became A Work Agent · p8 — Both state that frontier model capability is converging closely enough that competitive advantage (or, here, the locus of engineering difficulty) is migrating from the model layer to the harness/context layer.
supportsGitHub - musistudio/claude-code-router: Use Claude Code as the foundation for coding infrastructure, allowing you to decide how to interact with the model while enjoying updates from Anthropic. · p11 — Both frame the model as a swappable, increasingly commoditized component while the durable engineering value sits in the harness that wraps it.
explicitFor teams adding LLMs to incident response, effort spent preparing context pays off more than reaching for a larger model.conf: high
explicitdevelopedDeterministic pipelines that correlate signals up front and hand the model one fixed context let a wrong answer be attributed unambiguously to the model, whereas agent-based designs (and their bespoke tool-fetching choices) confound reasoning failure with harness failure.conf: high
supportsBuilding an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident · p6 — Both find that a deterministic, non-agentic preprocessing layer -- not model reasoning -- does the load-bearing work in AI-assisted incident response; the copilot's shipped tool is deterministic pattern-matching precisely because that layer is what actually matters.
extends ←Presentation: From Thousands to One: Building LLM-Powered Selection Systems · p9 — Extends the RCA article's claim that deterministic pipelines enable unambiguous model-vs-harness attribution: here, separating the LLM's textual step from deterministic code plus a discriminator call similarly yields clean, database-comparable per-decision accuracy signals in a structured-selection (not incident-diagnosis) domain.
extends ←Presentation: Decision Models in Agentic Architectures: From Production to Agent Skills · p8 — Both apply the same architecture: route the model's job to a deterministic component so failure can be attributed cleanly, rather than confounding reasoning failure with the agent/harness — Porcelli's case is a decision engine tool call rather than a pre-correlated diagnostic pipeline.
explicitIn a single controlled experiment -- a Chaos Mesh NetworkChaos delay injected between a catalogue service and its Postgres database, with the context deliberately salted with misleading signals (RTT-inflated query timings), run as one ~9,800-token prompt against eleven models -- closed frontier models (Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro) passed, larger open-weight models mostly kept pace, and lower down Gemma 4 31B was the only self-hostable model to identify the root cause, while the larger Qwen3.6 35B and Qwen3 Coder Next both missed it.conf: high
explicitAgent-based designs retain a genuine advantage over deterministic pipelines for novel incidents outside any predefined correlation set, because a model that fetches its own data can pull signals a fixed pipeline never anticipated.conf: high
extendsYour AI agent just blamed the network team. Now what? · p1 — This article's concession that agent-based designs keep an advantage for novel, unanticipated incidents supplies the missing trade-off analysis for the autonomous cross-domain diagnostic agent described in the other extract, which asserts agentic RCA capability without weighing it against deterministic alternatives.
explicitMulti-agent LLM incident investigations are notoriously hard to debug and operate in production because a failed run leaves no clean stack trace -- only unpredictable prompt interactions and emergent multi-agent coordination.conf: medium
explicitcorroboratedPractitioners are increasingly abandoning fully agentic incident-response designs in favor of mostly deterministic workflows with a narrow LLM step, citing better reliability and lower token cost.conf: medium
supports ←Presentation: From Thousands to One: Building LLM-Powered Selection Systems · p10 — Independently arrives, in a different domain (mapping free text to database IDs vs. SRE root-cause analysis), at the same architectural conclusion the RCA article reports as an industry trend: replace end-to-end agentic decision-making with deterministic narrowing plus a narrow, bounded LLM step.
explicitThe per-call cost of AI RCA is negligible (a few cents, even on frontier models) once the correlation work happens deterministically before the model is invoked.conf: high
implicitBecause Coroot tested only one experimental scenario (one fault type, one set of anticipated confounds), the conclusion that reasoning is 'basically solved' generalizes a result from an in-domain, pre-anticipated case to AI RCA as a whole, without evidence about how models or the harness perform on incident types the correlation pipeline wasn't designed to expect.conf: medium

Arguments

The harness -- the pipeline that assembles and correlates context before the model is called -- rather than the model itself, is now the binding constraint on AI RCA quality, so engineering effort should shift there. strength: medium

Explicit premises:

Reasoning was uniformly strong across most frontier and larger open-weight models when given the same well-prepared, single context.
A deterministic pipeline design lets you isolate whether a wrong answer came from the model or from the harness, unlike an agent-based design.
The correlation/preparation work, done deterministically, is what filters out (or fails to filter out) misleading signals before the model ever sees them.

Implicit premises:

The single Coroot benchmark scenario (one Chaos Mesh network-delay fault, with confounds the harness was built to anticipate) is representative of AI RCA difficulty in production generally.
Passing one designed scenario, with confounds the experimenter already knew about and deliberately included, is evidence that reasoning 'is basically solved' rather than evidence that this specific, anticipated confound was solved.

Inferences:

If harness quality dominates outcomes, then organizations facing AI RCA quality problems should audit and invest in their correlation/context pipeline before assuming they need a more capable (or more expensive) model -- the article draws this inference explicitly as its 'practical takeaway'.
The same reframing (separate the harness variable from the model variable to make failures diagnosable) likely generalizes beyond RCA to any AI-agent task where deterministic pre-processing is feasible.

Narratives

Coroot engineer Nikolay Sivko built one controlled scenario to separate 'can the model reason?' from 'does the harness feed it the right evidence?': a Chaos Mesh NetworkChaos experiment injecting delay between a catalogue service and its Postgres database, slowing queries and pushing the front-end into 502 errors, with the context deliberately including misleading signals (query timings inflated by network round-trip time). He ran the same ~9,800-token prompt -- asking for root cause, the cause-and-effect chain, and the immediate fix -- against eleven different models. Closed frontier models and most larger open-weight models passed (correctly naming the injected Chaos Mesh experiment and flagging the need to delete it and its schedule); lower down, Gemma 4 31B was the only self-hostable model to identify the root cause, while the larger Qwen3.6 35B and Qwen3 Coder Next both missed it.
→ This is the evidentiary backbone for the entire article -- it operationalizes the abstract claim 'reasoning is no longer the bottleneck' as a concrete, falsifiable test, and its pass/fail pattern across eleven models is what licenses the article's unity claim (propositions p1, p3, p4).

Questions addressed

✅ Is AI capable of doing root cause analysis? ("Can AI do RCA?")
Reframed rather than answered directly: Sivko argues this is 'the wrong question' because it conflates two separately measurable jobs (model reasoning vs. harness/context preparation); once separated, the article concludes model reasoning is largely adequate and the harness is the real open problem.
⭕ Which architecture -- agent-based (tool-fetching) or deterministic (pre-correlated, single-context) -- is better for AI RCA?
The article presents a genuine trade-off (agent-based flexibility for novel incidents vs. deterministic repeatability, debuggability and cost) without declaring an overall winner, though it notes an industry drift toward deterministic designs and ends by treating harness/context work as the shared, higher-leverage lever regardless of architecture.
⭕ Whether a hybrid architecture -- deterministic by default, with an agent-based fallback for incidents outside the predefined correlation set -- would combine the reliability of one camp with the coverage of the other.

Gaps

incompleteThe 'reasoning is basically solved' verdict rests on a single experimental scenario (one Chaos Mesh network-delay fault type, with one set of misleading signals the harness designer already anticipated). There is no report of repeated trials of the same scenario (to check consistency), no second incident type, and no test of an incident whose confounds were not anticipated at harness-design time -- exactly the case where the article itself says agent-based designs would have an edge.
Read next: Look for a multi-scenario or multi-trial replication of Coroot's benchmark, or for evaluation frameworks purpose-built to separate model-reasoning-quality from harness-quality across a broader set of incident types (e.g. AIOps/observability benchmark suites).
incompleteNo false-positive or confidence-calibration analysis is given. The experiment reports pass/fail on naming the correct root cause, but not what any model said when it was wrong (confidently wrong vs. uncertain), nor how often a model would confidently misdiagnose when the deterministic harness itself missed or mis-weighted a signal it wasn't built to anticipate -- the precise failure mode the article's own 'novel incidents' caveat implies is possible.
Read next: Look for incident-response AI evaluations that report calibration/false-positive rates, not just accuracy, especially under harness-blind-spot conditions.
incompleteThe article reports, without comment, an inverse-scaling result at the sub-frontier tier: Gemma 4 31B -- the smallest self-hostable model named -- was the only one of the three to identify the root cause, while the larger Qwen3.6 35B and Qwen3 Coder Next both missed it on the identical, well-prepared context. Sivko's 'basically solved' verdict reads as scoped to the frontier tier, and the piece does report this split openly rather than hiding it, but it never asks why a smaller model outperformed larger peers given the same harness -- training data, instruction-following style, or some other model-level factor the harness explanation can't account for.
Read next: Look for follow-up analysis of why Gemma 4 31B outperformed larger open-weight peers on this specific task, and for other reported cases of inverse scaling on well-prepared-context reasoning tasks.