Don't Let AI Make Bad Analytics Worse

Steven Tadelis · article · 2026-07-27 source ↗ Reader 🖥️

practical · argumentative — The author argues for an organizational operating model (a six-part 'decision-grade analytics' discipline) that companies should adopt before scaling AI analytics — every section builds toward specific managerial actions, not toward a descriptive theory of analytics.

Structure

Key terms

• self-service analytics
  An analytics model that starts with the user's question ('what do you want to know?') and optimizes for giving people fast access to answers.
• decision-service analytics Decision-Grade Analytics
  An analytics model that starts one step earlier than self-service — 'what decision are you trying to improve?' — and designs the measurement (comparison, causal question, experiment) around that decision.
• decision-grade Decision-Grade Analytics
  A standard analytics output should meet before it is allowed to influence a real decision — comparable to 'investment-grade' as a quality threshold rather than a binary correct/incorrect label.
• labeling the question Decision-Grade Analytics
  Explicitly classifying an analytics output as descriptive (what happened), predictive (what is likely to happen), or causal (what will happen if we intervene) before treating it as evidence for a decision.
• incrementality Decision-Grade Analytics
  The portion of an observed outcome (e.g., a purchase) that would not have happened without the intervention (e.g., an ad) — the object a causal, counterfactual analysis is trying to isolate.

Propositions

• explicitFraming AI analytics as an access problem ('let more people ask more questions') is the wrong starting point; the right starting point is helping people make better decisions with the answers they get.conf: high
• explicitdevelopedAs AI analytics scales, companies sort into three tiers by what they optimize for: most use it for efficiency (more automated queries, faster turnaround), better companies use it for reliability (semantic layers, documentation, provenance, validation), and the best use it for decision quality (redesigning analytics around the decisions that matter).conf: high
  extends11) AI-spesial med Vilde Opsal og Anders Sveen · p5 — Extends the general 'AI is a force multiplier in both directions, strong get stronger / weak get weaker' amplifier thesis into a specific domain (analytics), naming the three tiers (efficiency/reliability/decision-quality) that separate weak from strong organizational responses.
  extends ←Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents · p1 — Grab's reported metrics (mechanical-ticket share and self-service resolution rate) are exactly the kind of efficiency-tier gains the HBR piece says most companies optimize for; Grab's article gives a concrete real-world instance of that tier, plus claims (unverified here) of also investing in the reliability tier via certified data/ContextIQ.
• explicitCognitive tools like AI substitute for implementation skill while increasing the value of judgment, but in analytics specifically the pivotal managerial question is which organizational routines convert judgment into decision-grade evidence.conf: high
• explicitBefore asking AI to analyze data, leaders should specify the decision the analysis is meant to improve, what action might follow, and what evidence would change their mind — without this discipline, analytics produces interesting facts or motivated reasoning with better charts.conf: high
• explicitcorroboratedDescriptive, predictive, and causal questions require different standards of evidence, and AI systems tend to blur these distinctions by answering a descriptive or predictive question in a way that gets treated as causal evidence.conf: high
  supports ←Netflix Open-Sources Agentic Workflow for Causal Inference · p3 — Netflix's target-trial-emulation framing and critic checks (placebo test, adopter bias) are an applied instance of holding causal claims to a causal-appropriate evidentiary standard, rather than blurring it with descriptive/predictive confidence.
  extends ←Research: What Physical Retail Stores Deliver in a Digital Economy · p3 — Applies the descriptive/predictive/causal rigor principle concretely to retail investment: insists that a store's post-opening sales figures be tested via matched-comparison counterfactual before being read as caused by the store, rather than treated as causal at face value.
• explicitGood analysis exposes its own assumptions and counterfactuals by default; every AI-generated answer should carry a 'reason to trust this' and a 'reason to be cautious.'conf: high
• explicitcorroboratedSelf-service analytics increases, rather than decreases, the need for metric governance, because ambiguous metric definitions (e.g., what counts as 'active user' or 'churn') cannot be resolved by a model.conf: high
  supportsPresentation: Autonomous Data Products for the Autonomous Era: Rethinking Data Architecture for GenAI · p9 — Both converge on the same governance principle from different angles — Tadelis's Airbnb Minerva example (single source of truth for business metric definitions) parallels the data-products piece's claim that domain teams should own semantic definitions while a shared layer enforces consistency.
  supports ←Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents · p3 — Grab's ContextIQ and 5,000+ certified tables/metrics is a concrete instance of the metric-governance investment the HBR piece argues self-service analytics makes more necessary, not less.
• explicitOrganizations should triage analytics by stakes, the way banks apply risk-based model-risk-management: low-risk questions can move fast, but claims about budgets, pricing, personnel, or regulatory filings need review by people with relevant judgment before they influence decisions.conf: high
• explicitcorroboratedIf organizations reward teams for proving their initiatives worked and punish leaders who bring bad news, both human analysts and AI systems will be steered toward findings that justify decisions already made rather than toward truth.conf: high
  supports8) Christian Printzell Halvorsen om å lede på riktig nivå: Lokalt forankret, nordisk samlet · p12 — Both propositions argue that honest reporting (of trouble, or of disconfirming analytics findings) requires an organizational guarantee that admitting bad news brings help/correction rather than blame or persuasion pressure.
  supports ←Research: What Physical Retail Stores Deliver in a Digital Economy · p2 — The channel-P&L 'defend your own revenue' dynamic is a structural analog of the incentive-driven distortion described there: local success metrics push both stores and e-commerce toward framings that protect their own budget rather than surface the true causal picture.
• explicit"Data-driven" is a misleading phrase because data does not drive decisions — people do; data can discipline judgment but organizations can also confuse measurement with understanding.conf: high
• explicitcorroboratedThe right managerial response to abundant AI-generated answers is not to slow AI down everywhere, but to add friction selectively where the cost of being wrong is high while letting low-risk analysis move fast.conf: high
  supports ←Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents · p2 — Grab's staged autonomy levels (human review at defined gates for Levels 3-4, escalation rules for Level 5) and Scarlet's escalate-on-uncovered-cases design are a concrete instance of the HBR piece's prescription to add friction selectively by stakes rather than uniformly.
• explicitSome AI-generated analytics answers will be technically accurate but useless for the decision at hand, and some will be seductive precisely because they confirm what powerful people already believe.conf: high
  supportsThe Founder's Playbook: Building An AI-Native Startup · p10 — Both describe the same failure mode — AI outputs that are 'seductive because they confirm what powerful people already believe' (Tadelis) is the analytics-specific instance of 'AI as a research/ideation tool amplifies confirmation bias by default' (founders-playbook).
• implicitcorroboratedAn analytics operating model can only function if leaders, analytics teams, and senior executives each own a distinct piece of it (decisions, measurement discipline, incentives respectively); if any one group defaults, the other two cannot compensate.conf: medium
  supports ←Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents · p2 — Grab's stated retention of human accountability for metric definitions, causal interpretation, business assumptions, and final decisions maps onto the HBR piece's claim that an analytics operating model needs each accountability piece owned by a distinct party.
• implicitOrganizations that are already analytically undisciplined (no decision-first framing, no metric governance, no honest-feedback incentives) will see AI make their analytics worse, not better, once it removes the friction that previously slowed bad analysis down.conf: high

Arguments

Narratives

• eBay's paid-search branded-keyword advertising looked highly effective by standard metrics, until Tadelis and colleagues tested the counterfactual with holdout experiments and found much of the apparent value was not incremental — after which eBay made counterfactual testing central to evaluating marketing spend.
  → Concretely demonstrates the difference between a descriptive/correlational metric ('branded search converts well') and a causal, counterfactual-tested one ('branded search caused incremental purchases') — grounding the abstract 'label the question' and 'expose counterfactuals' principles in a specific, high-stakes real-world reversal.
• Airbnb, as it scaled, found different teams giving different answers to basic business questions because of inconsistent tables and metric definitions, and built Minerva as a single-source-of-truth metrics platform.
  → Supports the claim that self-service analytics increases (not decreases) the need for metric governance — shows what happens when democratized answers proceed without matching definitional discipline.
• Amazon's Working Backwards process (writing a mock press release/FAQ before building a product) and its 'Have Backbone; Disagree and Commit' leadership principle, both drawn from the author's personal experience there.
  → The Working Backwards story supports 'start with the decision' (forcing clarity before debating evidence); the Disagree and Commit story supports 'reward truth-seeking' (people were expected, not merely permitted, to challenge a leader's reading of the evidence).

Questions addressed

• ✅ Why doesn't giving more people AI-powered access to data analysis automatically produce better business decisions?
  Because access and decision quality are different axes; without decision-first framing, question-type labeling, exposed assumptions, metric governance, risk-based validation, and truth-seeking incentives, more answers just means more (possibly wrong or irrelevant) answers.
• ⭕ How should an organization decide which AI-generated analytics claims need human validation before acting, versus which can be trusted at face value?
  The article proposes the model-risk-management analogy (triage by stakes: budget, pricing, personnel, regulatory) but does not specify the actual mechanics of the triage — who decides stakes in advance, how disputes about classification get resolved, or how to catch high-stakes answers that arrive informally rather than through a designated review channel.
• ⭕ Whether the six-part operating model works as a bundle or whether some parts matter more than others, and in what sequence a company without any of them should adopt them.
  Not addressed — the six parts are presented as a flat, unordered list with no discussion of dependencies, sequencing, or relative leverage.

Gaps

• incompleteThe article never specifies how an organization identifies a 'high-stakes' analysis before seeing its answer. Real decisions often become high-stakes only in retrospect (an innocuous-looking metric turns out to justify a major reallocation once a VP notices it), so a triage gate defined purely by topic (budgets, pricing, personnel, regulatory) will miss high-stakes claims that arrive through informal channels — a Slack-shared chart, a throwaway dashboard query — before any formal review point exists.
  Read next: Model-risk-management literature on how banks classify model materiality and handle 'shadow models' that escape formal inventories would show whether this same blind spot appears in finance and how (or whether) it gets closed there.
• incompleteThe piece asserts that AI can 'blur' the descriptive/predictive/causal distinction but does not explain the mechanism by which it does so — is it that AI models are trained on correlational data and cannot distinguish causal from correlational patterns without explicit experimental data, or is it purely a communication/framing failure by users who ask loosely worded questions? These have different fixes (better model training and disclosure vs. better user education and question templates).
  Read next: Causal-inference-in-ML literature (e.g., work on distinguishing observational from interventional/counterfactual queries in LLM-based analytics assistants) would clarify whether this is a solvable interface problem or a deeper representational limitation.
• incompleteThe six-part model is presented as equally applicable across company sizes and maturities, but the case narratives (Amazon, eBay, Airbnb) are all large, resource-rich organizations that could build dedicated platforms (Minerva) or run large-scale randomized experiments. The article doesn't address whether a small company without a data-science team or a marketing-experimentation budget can implement 'expose counterfactuals' or 'govern the metrics' at any meaningful fidelity, or what a minimum-viable version looks like.
  Read next: Practitioner writing on lightweight/low-cost causal inference and metrics governance for resource-constrained teams would test whether the model degrades gracefully or requires the scale these examples assume.

Related documents

• The Founder's Playbook: Building An AI-Native Startup
• Presentation: Autonomous Data Products for the Autonomous Era: Rethinking Data Architecture for GenAI
• 8) Christian Printzell Halvorsen om å lede på riktig nivå: Lokalt forankret, nordisk samlet
• 11) AI-spesial med Vilde Opsal og Anders Sveen
• The "funhouse mirror": How AI reflects the hidden truths of your software pipeline
  AI as Amplifier Guardrails Human-in-the-Loop
• Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents
  Decision-Grade Analytics Guardrails Human-in-the-Loop
• Looking back at the Future of Software Engineering unconference
  AI as Amplifier Guardrails Human-in-the-Loop
• 11 Tips For AI Coding With Ralph Wiggum
  Guardrails Human-in-the-Loop
• Article: Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules
  Guardrails Human-in-the-Loop
• Your AI agent just blamed the network team. Now what?
  Guardrails Human-in-the-Loop