Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident

Sudhakararao Sajja · article · 2026-07-07 source ↗ Reader 🖥️

practical · build log / how-to — a first-person 'how I built it' walkthrough of an open-source CLI (incopilot): architecture, module pipeline, setup commands, and design-decision rationale, ending with a roadmap — A ~820-word single-practitioner build log for an open-source Python tool that automates first-pass incident triage. It is prescriptive and rationale-driven ('why Python', 'why read-only', 'why local'), grounded in the author's own on-call experience rather than in measured outcomes — the test of every claim is 'does this get you to diagnosis faster', asserted, not demonstrated. The 'AI' framing is largely aspirational: the shipped version is deterministic pattern-matching, with LLM summarization named only as a future roadmap item.

Structure

Key terms

• incident copilot (incopilot)
  An open-source Python CLI that automates the entire first-pass triage of a production incident — collecting logs, detecting failure patterns, and generating an incident brief — so the engineer skips the manual search and goes straight to problem-solving. 'Copilot' here means an assistant that prepares context, not one that flies the plane: it never diagnoses or acts.
• the first 15 minutes
  The opening phase of every incident, before real diagnosis begins — a 15-to-45-minute manual loop of pulling logs and guessing what to check. The author's key move is to classify this phase as searching, not debugging: repetitive, low-judgment work that is 'largely automatable'.
• read-only (design constraint)
  The deliberate rule that the tool never writes to the system, restarts services, or executes state-changing commands — enforced via a config-level safe-command list. Framed as a feature, not a limitation: it makes the tool safe to run during an active incident and usable by any on-call engineer regardless of system familiarity, because it 'cannot make a bad incident worse'.
• collectors / analyzer / reporter (the module pipeline)
  The three-stage coined pipeline: collectors pull logs from systemd journal, Docker containers, or raw files; the analyzer detects high-signal events (OOM kills, 5xx spikes, disk exhaustion, network timeouts, service restarts) and normalizes lines; the reporter maps findings to the Four Golden Signals and emits a structured brief in markdown and JSON.
• Four Golden Signals (latency, traffic, errors, saturation)
  The Google-SRE monitoring taxonomy the reporter maps every detected failure pattern onto, giving the brief a familiar, structured shape rather than a raw log dump.

Propositions

• explicitThe first phase of any incident is not debugging but searching — you don't yet know what broke, which service is responsible, or where to look — and this search work is repetitive, stressful, low-judgment, and 'largely automatable', costing 15-45 minutes per incident and thousands of engineering hours per year across teams.conf: high
• explicitdevelopedThe tool is designed to sit strictly on the data-gathering side of the human/automation boundary: it collects logs, detects high-signal patterns, maps them to the Four Golden Signals, lists the specific triggering log lines, and assembles a structured brief — and is deliberately read-only, never writing, restarting, or executing state-changing commands — so the responder's diagnosis and every corrective action remain entirely human.conf: high
  supportsPresentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) · p16 — Reed's human-factors finding is that AI used as explanation-only / data-gathering / attention-redirection outperforms AI used to make recommendations, especially as task complexity rises. incopilot's design is a concrete instance of that prescribed pole: it is read-only, it assembles evidence and redirects attention (which log lines fired, which Golden Signal), and it never prescribes an action or takes one — diagnosis and remediation stay human. The tool's boundary placement is exactly what Reed argues is the safe way to put automation into incidents.
  refines11 Tips For AI Coding With Ralph Wiggum · p14 — Pocock's Ralph technique presupposes that a feedback loop plus explicit scope can SUBSTITUTE for human judgment on a bounded task, relocating the human to up-front scoping and after-the-fact verification. incopilot draws the human/automation boundary more conservatively: rather than substituting for judgment, it places the boundary BELOW judgment entirely — automating only the mechanical, pre-diagnostic search, structurally forbidding the tool from acting (read-only), and leaving both the diagnosis and every action with the human. Same underlying design question (where does automation stop and human judgment begin), a stricter and safer answer for a high-consequence, non-deterministic context.
  extends ←Your AI agent just blamed the network team. Now what? · p1 — Extends a single-domain, deterministic, strictly read-only log-search tool's human/automation boundary into a genuinely autonomous, cross-domain, judgment-bearing agent -- introducing the organizational-political stakes (team-implicating findings) entirely absent from the narrower tool.
• explicitcontestedBecause 'logs contain the answer almost every time' and the only problem is that engineers spend the first 15 minutes manually finding where it is buried, an automated tool can reliably run that search 'in under 60 seconds' and surface the likely causes / 'what the evidence suggests'.conf: high
  contradictsPresentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) · p15 — The copilot's load-bearing optimism — 'logs contain the answer almost every time', the search reliably done 'in under 60 seconds', likely causes surfaced fast — runs against Reed's asymmetric-risk finding: when automation's pattern-surfacing is accurate the upside is moderate (+53-67%), but when it is misleading the downside is catastrophic (-96-120%), and operators cannot detect in real time which regime they are in. The build log presents fast pattern-surfacing as an unalloyed benefit with no false-positive analysis; Reed shows that in incidents specifically, a confident-but-wrong brief is not a smaller version of a right one — it is where the damage lives (see gap g2).
• explicitThe value of the tool is that responders start from a structured brief instead of a blank page — the report.md is formatted to paste straight into an incident doc, Slack, or PagerDuty, and report.json to POST to a webhook or attach to a ticket — removing not just time but cognitive load and the compounding pressure of an active incident at 2am.conf: high
• explicitLocal, self-contained execution — running on the host or inside the container with no external API calls and no data leaving the environment — is a first-class design requirement, justified by data-residency requirements and security-sensitive production environments.conf: high
• explicitcorroboratedThe shipped tool is deterministic pattern-matching, not AI; the 'AI incident copilot' framing is forward-looking, with LLM summarization named as a future roadmap layer and Python chosen specifically to make that later transition straightforward.conf: high
  supports ←AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering · p3 — Both find that a deterministic, non-agentic preprocessing layer -- not model reasoning -- does the load-bearing work in AI-assisted incident response; the copilot's shipped tool is deterministic pattern-matching precisely because that layer is what actually matters.
  extends ←Expedia Uses AI Driven Service Telemetry Analyzer to Accelerate Incident Investigation · p2 — The sibling tool ships only deterministic pattern-matching today and names LLM summarization as a future roadmap item ('the AI incident copilot framing is forward-looking'). STAR is a further step down that same trajectory already in production: a deterministic, non-agentic workflow that already incorporates real LLM analysis (domain-specific prompts, multi-stage summarization) rather than pattern-matching alone.
• implicitcontestedThe early-triage 'search' phase is cleanly separable from and genuinely lower in judgment than the 'diagnosis' phase, so automating the former neither pre-empts nor degrades the latter — the boundary between mechanical context-assembly and human judgment is treated as sharp and safe.conf: high
  contradicts ←This Week in AI: When agents outnumber people · p8 — This piece's unexamined assumption that people will keep possessing the foundational knowledge needed to catch AI errors is directly undercut by the other document's claim that automating the routine, judgment-building practice does not degrade the higher-judgment work built on top of it — that is precisely the assumption this document's implicit premise leaves untested.
• implicitFaster, automated context assembly reliably improves incident outcomes — a brief in 60 seconds is assumed to be strictly better than a blank page — with no consideration that a fast but wrong brief could point responders in the wrong direction or that time-to-brief is not the same as time-to-resolution.conf: high

Arguments

Narratives

• An alert fires at 2am. An engineer wakes up, SSHs into a server, and begins the manual loop — pulling logs, scanning for errors, guessing what to check next — for 15 to 45 minutes before real diagnosis even begins. The author has 'been on that on-call rotation' and knows what it costs: time, cognitive load, missed context, and the compounding pressure of an active incident.
  → The motivating pain narrative and the author's credibility claim; it defines the toil the tool removes and frames the entire build as scratching a personal on-call itch.

Questions addressed

• ✅ Why are the first 15 minutes of an incident the hardest, and why are they automatable?
  Because that phase is searching, not debugging — you don't yet know what broke or where to look — and search across large volumes of logs (pattern recognition, signal extraction, structured summarization) is exactly the kind of work automation handles well.
• ✅ How do you make an automated tool safe to run during an active incident and usable by any on-call engineer?
  Make it deliberately read-only (never write, restart, or execute state-changing commands, enforced by a safe-command list) and run it locally with no external API calls — so it cannot make a bad incident worse and cannot leak data.
• ⭕ Does automating the first-pass triage actually reduce mean-time-to-resolution or improve incident outcomes?
  Unaddressed. The piece quantifies only the toil removed (15-45 minutes of search, thousands of engineering hours) and asserts responders 'skip straight to actual problem-solving', but offers no measurement that a faster brief shortens resolution or improves outcomes.
• ⭕ How often does the pattern detection surface the wrong likely cause, and what does anchoring on a misleading brief cost?
  Unaddressed. There is no accuracy, precision, or false-positive analysis, and no discussion of the risk that a confident-looking but wrong brief steers a responder in the wrong direction.

Gaps

• incompleteNo production evidence that the tool reduces MTTR or improves incident outcomes. The entire benefit case rests on toil removed (15-45 minutes of manual search, 'thousands of engineering hours lost per year') plus the assertion that responders can then 'skip straight to actual problem-solving' — but time-to-brief is not time-to-resolution, and nothing here is a controlled before/after or a field measurement. For a rigorous adopter this is the first missing thing: a build log's plausible mechanism, not demonstrated impact.
  Read next: Incident-response outcome measurement: DORA's failed-deployment-recovery / MTTR data, controlled before-after studies of triage-automation on resolution time, and whether pre-assembled context actually shortens the diagnosis phase or just relocates the clock.
• incompleteNo evaluation or false-positive analysis of the pattern detection. The tool surfaces 'likely causes' and 'what the evidence suggests', but there is no accuracy/precision/recall figure, no false-positive-rate discussion, and no account of the downside case — a fast, confident, but wrong brief. This matters acutely for incident automation specifically: the human-factors literature finds the risk is asymmetric — automation that surfaces misleading patterns can make outcomes far worse than no automation, and responders cannot detect in real time which regime they are in — precisely the analysis a single-practitioner build log omits.
  Read next: Read J. Paul Reed on the ironies of automation and AI in incidents (the asymmetric-risk / nurses-study finding), plus eval methodology for automated triage/alerting: precision/recall, false-positive cost, and joint human-AI testing rather than testing the detector in isolation.
• incompleteThe deskilling and automation-complacency risk is unexamined. Automating the first-pass search is exactly the routine practice that builds the diagnostic intuition responders need when the tool is wrong, absent, or facing a novel failure; a read-only brief that confidently names likely causes can also anchor a responder on the tool's hypothesis. The piece treats the search/diagnosis boundary as clean and cost-free, never asking what erodes when the mechanical part is permanently delegated.
  Read next: Bainbridge's ironies of automation (skill decay from disuse, capability-induced complacency), and how incident-response teams retain diagnostic mental models while automating context-gathering — joint cognitive systems and skill-retention practice.
  closed byHuman Oversight Is Not a Human in the Loop — Bataller's Bainbridge-irony argument (p4) explains precisely why delegating routine first-pass diagnostic work erodes the intuition needed for rare/novel failures, and the institutional-competence-maintenance claim (p15) extends it: organizations must deliberately design to preserve practice opportunities rather than assume the deskilling risk resolves itself.

Related documents

• Your AI agent just blamed the network team. Now what?
  Guardrails Human-in-the-Loop SRE & Incident Response
• Expedia Uses AI Driven Service Telemetry Analyzer to Accelerate Incident Investigation
  Human-in-the-Loop SRE & Incident Response AI Workflow Patterns
• How Figma Uses AI Agents for Security
  Guardrails Human-in-the-Loop SRE & Incident Response
• Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents
  Guardrails Human-in-the-Loop SRE & Incident Response
• Stop Automating Old Processes. Design New Ones Instead.
  Guardrails Human-in-the-Loop AI Workflow Patterns
• What is “loop engineering?”
  Human-in-the-Loop SRE & Incident Response AI Workflow Patterns
• What Survived Contact
  Data Sovereignty Guardrails Human-in-the-Loop
• 11 Tips For AI Coding With Ralph Wiggum
  Guardrails Human-in-the-Loop
• Article: Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules
  Guardrails Human-in-the-Loop
• Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation
  Guardrails Human-in-the-Loop