Is Your AI Training the Competition?

William Marks · article · 2026-09-08 source ↗ Reader 🖥️

AI Industry Litigation Enterprise AI Adoption & Readiness Knowledge Distillation (Teacher-Student Compression)Moats & Defensibility Organizational Legibility

Beyond protecting raw data, companies must actively defend 'logic sovereignty' — ownership of the specific expertise and reasoning their employees apply through AI tools — because AI providers absorb that expertise through feedback-clause carve-outs, synthetic-training pipelines, and the structural flywheel of heavy use, in ways standard 'no training on our prompts' contracts do not prevent.

practical · mixed — Primarily practical/argumentative — each of the three 'channels' ends in an explicit 'What managers should do' prescription — but built on an expository reframing (data vs. logic, AI Sherlocking, logic sovereignty) and grounded with narrative case evidence (Apple/Watson, Anthropic's Feb 2026 disclosure, Samsung's 2023 leak).

Structure

Part	Role
Reframing the risk question, via the Apple/Watson 'Sherlocking' origin story	context / definitional setup
How AI Sherlocking works: it targets the reasoning behind an answer, not a copyable feature; sandboxing stops data leaks but not pattern-learning	mechanism / evidence
Anthropic's February 2026 distillation-attack disclosure (DeepSeek/Moonshot/MiniMax)	concrete evidence, adversarial extreme case of the general mechanism
Shadow AI: Samsung's 2023 ChatGPT leak, the 2025 MIT adoption study, IBM's 2025 breach-cost data	concrete evidence, mundane/human channel for the same loss
'Logic sovereignty' as the successor framing to 'data is the new oil'	conceptual synthesis — restates the unity in named, portable form
Three Takeaways, each mapping one channel (feedback carve-outs; synthetic-data reuse; platform-learns-from-best-users) to a specific managerial fix	structured argument plus prescriptive payoff
Closing reframe ('who is learning from our success?'), boardroom-topic prediction, and the industry-wide concentration stake	conclusion / stakes

Key terms

AI Sherlocking
An AI platform absorbing a company's hard-won, specific problem-solving logic through ordinary tool use — not by copying a visible feature, but by learning the reasoning pattern behind good answers — so that a competitor's next similar question is answered by a model already sharpened on your expertise.
logic sovereignty
Ownership not just of a company's data but of the thinking — the specific judgment, workflow, and reasoning — it applies to that data; the thing the author argues is now scarcer and more valuable than data itself.
distillation Knowledge Distillation (Teacher-Student Compression)
Using a stronger model's outputs to train a weaker one to imitate it — illustrated by the Anthropic disclosure, where three Chinese labs allegedly used fraudulent accounts to harvest Claude's answers at scale for this purpose.
shadow AI
Employees using free, personal AI accounts for work — out of habit, convenience, or to route around enterprise restrictions — so that whatever they paste in goes straight into an unprotected training pipeline with no enterprise agreement involved.

Propositions

explicitStandard enterprise AI contracts that protect prompts and outputs from training use are necessary but insufficient, because a company's most valuable AI-era asset — the expertise blended into how it uses the tool — leaks through channels those contracts don't cover.conf: high
explicitAI Sherlocking — a platform absorbing a company's problem-solving logic through ordinary use rather than copying a visible feature — is the AI-native successor to the pattern named after Apple's Sherlock app copying and killing Watson.conf: high
extendsAnette Mellbye's Post · p2 — Generalizes the 'platform absorbs wrapper functionality' dynamic (there, about B2B SaaS wrapper products) to a new target: any enterprise customer's internal problem-solving expertise, absorbed through ordinary use rather than feature-copying.
implicitcorroboratedWhat makes a given AI answer valuable is the blend of the model's general capability and the specific human expertise steering it — and that blend, not the raw prompt or output text, is what actually leaks when a provider learns from usage.conf: high
supports ←LLMs Reward Expertise · p1 — Both claims locate AI output value in the blend of model capability and the specific human expertise steering it, rather than in the raw prompt/output text itself.
explicitEmployee feedback to AI tools (ratings, corrections, 'try again') is contractually carved out as a separate category from prompts/outputs in both OpenAI's and Anthropic's standard terms, letting the vendor use and build on it freely and without payment even under an otherwise training-restricted enterprise agreement.conf: high
extendsYou Outsourced the AI—but You Still Own the Risk · p3 — That piece's 'Exposure 1' remedy is generic AI-contract transparency requirements (training-data categories, failure modes, etc.); this article identifies a specific additional contractual gap — the feedback carve-out clause — that a generic transparency requirement wouldn't close on its own.
explicitAnthropic's February 2026 disclosure that DeepSeek, Moonshot AI, and MiniMax ran more than 16 million distillation queries against Claude through roughly 24,000 fraudulent accounts demonstrates, in adversarial form, the same flywheel dynamic that every major provider runs legally and continuously across its paying customer base.conf: high
extendsA big week for AI denialism · p16 — That proposition states, abstractly, that whether training on a competitor's outputs is legally actionable IP theft remains open, which is why labs haven't sued distillers; this article supplies the concrete real-world instance (Anthropic's Feb 2026 disclosure, named labs, account/query counts) underlying that abstract claim.
implicitSandboxing AI tools protects data confidentiality but does nothing to stop a provider from learning a team's problem-solving patterns through ordinary interaction inside that sandbox.conf: high
explicitDespite most companies paying for secure enterprise AI accounts, employees at over 90% of surveyed companies (2025 MIT study) still regularly use personal AI tools for work, routing proprietary material into an unprotected training pipeline with no enterprise agreement in the way.conf: high
explicitBecause high-quality public training data is becoming scarce (Epoch AI projects substantial exhaustion 2026–2032), providers increasingly train next-generation models on their own systems' best outputs (e.g., Microsoft Phi-4's 400B synthetic tokens), which makes any AI output generated outside a walled-off enterprise contract fair game for the next model generation.conf: high
implicitcontestedBecause platforms improve by learning from their most sophisticated users, a company's own excellence at applying AI to a hard, high-value problem directly trains the platform to be better at that class of problem for every other customer, including direct competitors — inverting the usual relationship between being the best user of a tool and gaining advantage from it.conf: high
contradictsBaseten CEO Tuhin Srivastava on the AI Inference Crunch, Custom Models · p2 — Baseten's CEO argues the application layer survives because companies hold 'unique user signal that labs cannot access'; this article argues labs effectively do access it, indirectly, via the feedback/flywheel/distillation channels it documents — the signal is not as inaccessible as claimed.
explicit"Data is the new oil" is no longer sufficient competitive-strategy advice, because data has become abundant and commoditized while the scarce, valuable asset is now 'logic sovereignty' — ownership of the judgment, workflow, and reasoning applied to data, not the data itself.conf: high
explicitThe operative question for AI security reviews should shift from 'is our data safe?' to 'who is learning from our success?' — moving from a confidentiality lens to a competitive-capability lens.conf: high
implicitBecause platform-level absorption of user expertise happens automatically and continuously as a byproduct of ordinary, contractually-compliant use, no purely contractual fix can eliminate the exposure for a company's most strategically sensitive AI work — it can only be compartmentalized or reduced through deployment and workflow choices (private/on-prem models, task decomposition, in-house fine-tuning).conf: medium

Arguments

Standard 'no training on our prompts' contract protections do not actually protect a company's most valuable AI-era asset. strength: high

Explicit premises:

Enterprise contracts typically cover only the literal prompt and the raw model output.
Feedback (ratings, corrections) is contractually carved out as a separate category the vendor may use freely.
Providers also train future models on their own systems' outputs at scale (synthetic data) and on the aggregate pattern of similar high-quality work happening across the user base.

Implicit premises:

The value AI Sherlocking targets is the blend of model capability and applied human expertise, not the literal prompt/output text — so protecting only that text protects the wrong unit of value.
Providers have a structural business incentive (the 'flywheel') to keep learning from usage regardless of what any single contract clause says.

Inferences:

A company cannot eliminate this exposure through contract language alone; it must also change what kind of work it routes to which deployment tier.

AI Sherlocking is structurally different from, and harder to defend against than, traditional Sherlocking.

Explicit premises:

Traditional Sherlocking (Apple/Watson) copied a visible, discrete feature.
AI Sherlocking absorbs 'something you can't see: the reasoning that produced a good answer.'

Implicit premises:

Legal and technical defenses (patents, contract clauses naming specific data, sandboxing) are built to protect discrete, nameable artifacts — not a diffuse pattern learned across millions of interactions.
If the thing being absorbed has no discrete legal form, no defense keyed to discrete artifacts can fully stop it.

Inferences:

Patent and trade-secret law, built around identifiable IP, may be structurally mismatched to this threat — consistent with the corpus finding (extract-a-big-week-for-ai-denialism#p16) that labs have not been sued over distillation despite likely ToS violations, for lack of a clean legal hook.

Managers should treat AI feedback mechanisms as a contract-negotiation issue, not merely a product setting. strength: medium

Explicit premises:

OpenAI's and Anthropic's contracts both carve feedback out from 'no training' protections.
OpenAI's documentation states that even a user who opted out of training generally still contributes the whole associated conversation via a single thumbs-up/down.

Implicit premises:

Most legal/security reviews currently stop at confirming 'no training on our prompts' and treat that as sufficient — the argument's force depends on this being widespread executive complacency.

Inferences:

Vendor contract templates will likely need a newly negotiated clause category ('feedback and interaction-signal protection') as this awareness spreads — consistent with the article's own prediction that this becomes 'a standard boardroom topic.'

Being an AI vendor's most sophisticated user is a competitive liability rather than, as with ordinary tools, an advantage. strength: high

Explicit premises:

AI platforms improve by learning from their most sophisticated users.
The resulting improved capability is then sold to the whole market, including direct competitors.

Implicit premises:

The improvement transfers as general capability for a class of problem, not as raw data the company could control — so there is no way to license the resulting capability lift exclusively.
The company doing the teaching receives no compensation or exclusivity for the capability lift its usage produces.

Inferences:

Under this logic, deliberately staying an unsophisticated or intermittent AI user for the most strategic work becomes a rational, if uncomfortable, competitive strategy in some cases — an inference the article gestures at (task-splitting, in-house fine-tuning) without stating it this starkly.

Narratives

Apple built Sherlock, a search tool that copied the third-party Watson app feature-for-feature in the early 2000s and put its maker out of business almost overnight — coining 'getting Sherlocked.'
→ Establishes 'Sherlocking' as a familiar, named, retrospectively-obvious pattern of platform absorption before extending it to the harder-to-see AI version — the analogy does the persuasive work the article's abstract definition alone couldn't.
In February 2026, Anthropic disclosed that DeepSeek, Moonshot AI, and MiniMax ran over 16 million queries against Claude through roughly 24,000 fraudulent accounts to train competing models via distillation, identified by unusual concentration on Claude's most distinctive reasoning/coding capabilities.
→ Makes the abstract 'flywheel' mechanism concrete and undeniable by showing its adversarial extreme, licensing the pivot to 'the same underlying dynamic in a milder, entirely legal form' operating on every paying customer.
Samsung engineers inadvertently uploaded proprietary semiconductor source code to ChatGPT in 2023 while using it to debug code, prompting Samsung to ban the tool internally.
→ Grounds 'shadow AI' as an already-materialized incident rather than a hypothetical, and shows the loss occurring through the most mundane possible channel — ordinary employee habit, no sophisticated AI Sherlocking mechanism required.

Questions addressed

✅ Is my company's AI use protected from training the provider's future models?
Partially: literal prompts and outputs are typically protected under enterprise contracts, but feedback signals, shadow/non-enterprise use, and the structural capability transfer from heavy usage are not.
✅ What should a company concretely do differently?
Three channel-specific fixes: renegotiate contracts to explicitly cover feedback/ratings/interaction logs; move the highest-value work to private/on-prem deployments walled off from any training pipeline; decompose the most strategically sensitive tasks so the 'stitching' step that reveals the overall method stays in-house.
⭕ How much competitive damage does AI Sherlocking actually cause — to whom, how fast, and at what scale?
Never quantified; addressed only impressionistically ('gradually and then all at once').
⭕ Is this dynamic genuinely new, or a rebranding of the already well-understood 'data flywheel' / network-effects concept in tech strategy?
The author concedes in passing 'it's not entirely new' but never engages existing flywheel/network-effect literature to say what, specifically, is new about the AI version versus the pre-AI platform-economics pattern.

Gaps

incompleteThe article never quantifies how much of a platform's improvement on a class of problem is actually attributable to any single customer's usage versus the aggregate of millions of users solving similar problems. Without evidence isolating attribution, 'your best people are training your competitors' could be true only in a diffuse, per-company-negligible sense — a materially different risk than the article's framing implies.
Read next: ML research on marginal per-customer contribution / attribution in RLHF and continued pretraining; any published data on how concentrated vs. diffuse a platform's capability gains are across its customer base.
incompleteThe piece treats all AI use as equally exposed to AI Sherlocking, but never addresses whether exposure should scale with how genuinely novel a company's 'logic' actually is — a company solving well-known problems the standard way has little unique reasoning to lose, while a company with a truly novel approach has much more at stake. The article's advice (contract renegotiation, private deployment, task decomposition) is presented as universal rather than triaged by this distinction.
Read next: Literature on which capabilities are genuinely defensible versus commodity-grade (moats / adaptive-advantage corpus threads); frameworks for triaging AI use cases by IP sensitivity.
incompleteThe article's proposed defenses — splitting a task apart and keeping only the 'stitching together' step in-house, or using a public model for rough ideas and fine-tuning a smaller in-house model on the real work — are never evaluated for practicality: how much of the value of frontier-model reasoning is lost when the integrating step is withheld, and whether a smaller in-house model can absorb frontier-quality reasoning without becoming a materially weaker substitute.
Read next: Research on task-decomposition costs in AI-assisted workflows; small-model fine-tuning/distillation-from-a-public-model literature and its quality gap versus frontier models.
uninformedThe article notes Anthropic added feedback-management controls (Rate Chats toggle) for Team/Enterprise plans in March 2026 but explicitly flags it as 'not a tried and proven guarantee' — without examining what, technically, turning it off actually disables. It may stop only the explicit thumbs-up/down UI signal while implicit correction signals (edits, re-prompts, task abandonment/completion) remain uncontrolled and unaddressed by the toggle.
Read next: Independent technical verification or documentation of exactly what Anthropic's Rate Chats setting does and does not disable.

Closes gaps elsewhere

This Week in AI: Production Viability · g5 — That gap flagged a conflation of 'training data leakage' with 'prompt-time context use' as having very different IP-erosion profiles, undistinguished in the source piece. This article draws exactly that distinction explicitly: contract-protected prompt/output use (prompt-time, generally safe) versus feedback signals, synthetic-output reuse, and shadow-AI use (all genuine training-data leakage channels, each with its own mechanism and remedy).

Is Your AI Training the Competition?

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Closes gaps elsewhere

Related documents