Knowledge Distillation (Teacher-Student Compression)
Training a smaller, cheaper-to-serve "student" model to approximate the quality of one or more larger "teacher" models — the standard production technique for getting near-frontier ranking/relevance quality within real-time latency and cost budgets.
Also known as: knowledge distillation, distillation, teacher-student, teacher-student distillation, multi-teacher distillation, student model, teacher model, online distillation, offline distillation, distillation loss, model compression
Part of: Inference Optimization
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| Is Your AI Training the Competition? William Marks · 2026-09-08 | distillation | Using a stronger model's outputs to train a weaker one to imitate it — illustrated by the Anthropic disclosure, where three Chinese labs allegedly used fraudulent accounts to harvest Claude's answers at scale for this purpose. |
Documents (3)
- How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation Claudio Masolo · 2026-09-11 source ↗ Reader 🖥️
LinkedIn made multi-teacher knowledge distillation practical for its production job-search ranking system not through a distillation-algorithm breakthrough but through training-infrastructure engineering — a custom SGLang-based framework that serves teacher models directly inside the training loop (an online mode for fast iteration, an offline precomputed-and-cached mode once teachers stabilize) plus a stack of largely independent training-speed optimizations — collectively yielding a roughly eightfold training speedup that lets a compact 0.6B student model approach cross-encoder-quality ranking while avoiding the cost and latency of calling a frontier LLM on every search request. - Is Your AI Training the Competition? William Marks · 2026-09-08 source ↗ Reader 🖥️
Beyond protecting raw data, companies must actively defend 'logic sovereignty' — ownership of the specific expertise and reasoning their employees apply through AI tools — because AI providers absorb that expertise through feedback-clause carve-outs, synthetic-training pipelines, and the structural flywheel of heavy use, in ways standard 'no training on our prompts' contracts do not prevent. - Transcript for Cursor Team: Future of Programming with AI | Lex Fridman Podcast #447 Lex Fridman · 2024-10-06 source ↗ Reader 🖥️
AI coding tools should be built to maximize the human programmer's speed, control and agency over their code — not to replace programming with natural-language delegation to an autonomous agent — because the hardest and most valuable part of engineering is the dense sequence of small judgment calls that specifying intent upfront cannot capture, and because verifying AI-generated code is at least as hard a problem as generating it.
Propositions touching this concept (9)
By publication month, oldest first (extraction month when no publication date is known).
2024-02
- implicitThe own-model-filtering, two-stage training approach used here (predictions used only to filter/delete bad snippets, never as new training targets) is implied to be preferable to a teacher-student pseudo-labeling approach for reducing hallucinations, though no direct ablation against teacher-student retraining is reported within the paper.
2026-07
- implicitThe proxy model is task-specific knowledge distillation relocated into the database and triggered at query time — using a frontier model as teacher to train a cheaper student for a single predicate — rather than a general inference-serving optimization on the same model.
2026-09
- explicitAnthropic's February 2026 disclosure that DeepSeek, Moonshot AI, and MiniMax ran more than 16 million distillation queries against Claude through roughly 24,000 fraudulent accounts demonstrates, in adversarial form, the same flywheel dynamic that every major provider runs legally and continuously across its paying customer base.
- explicitdevelopedLinkedIn built a custom multi-teacher distillation framework on SGLang that serves teacher models directly inside the training loop, compressing knowledge from large teacher models into a compact 0.6B-parameter ranking model for AI-powered job search.
- explicitThe main contribution of this work is not the distillation technique itself but the systems engineering that makes distillation fast enough for rapid iteration.
- explicitServing teacher models for supervision during training becomes a bottleneck for a ranking system handling hundreds of thousands of queries per second, which motivates splitting the approach into online (live-query) and offline (precomputed/cached) multi-teacher distillation modes.
- explicitScaling Online Multi-teacher Distillation across multiple nodes with local teacher replicas accelerated the distillation process by 3x while keeping latency low enough for quick iterations.
- explicitOffline Multi-teacher Distillation — precomputing teacher outputs and storing them on HDFS or NFS instead of querying teachers live — reduces serving overhead and prevents repeated computation.
- explicitThe 0.6B student model, distilled from an 8B relevance oracle and a 1.7B engagement teacher, raised NDCG@10 for job search results by 24.48% (from 0.7583 to 0.9432).
Open gaps (4)
- incompleteThe article's proposed defenses — splitting a task apart and keeping only the 'stitching together' step in-house, or using a public model for rough ideas and fine-tuning a smaller in-house model on the real work — are never evaluated for practicality: how much of the value of frontier-model reasoning is lost when the integrating step is withheld, and whether a smaller in-house model can absorb frontier-quality reasoning without becoming a materially weaker substitute.Is Your AI Training the Competition? — read next: Research on task-decomposition costs in AI-assisted workflows; small-model fine-tuning/distillation-from-a-public-model literature and its quality gap versus frontier models.
- illogicalThe 'roughly eightfold' training speedup headline cannot be reconstructed from the individually cited multipliers (3x from the multi-node online-distillation architecture, 2x batch size from LiGer, up to 3.5x from multi-node training, +20% from FSDP2, up to 30% from H200 clusters) — naively multiplying them yields a number well above 8x, and the piece never shows which multipliers overlap, are additive rather than multiplicative, or apply to different, non-cumulative pipeline stages.How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation — read next: Read LinkedIn's original engineering blog post directly for the actual before/after wall-clock benchmark and the ablation methodology isolating each optimization's contribution.
- incompleteThe NDCG@10 improvement (0.7583 → 0.9432) and the throughput improvement (290 → 2,000+ items/sec/GPU) are attributed to a separate, only loosely specified piece of LinkedIn research (structured pruning and context compression), and it is unclear whether both figures come from the same deployed 0.6B student model, the same evaluation set, or the same production conditions.How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation — read next: The linked LinkedIn 'Reimagining LinkedIn's Search Stack' post for the full evaluation methodology and whether the quality and throughput figures share the same underlying system.
- incompleteNo discussion of model/quality drift or ongoing maintenance cost: as the teacher models themselves get updated or replaced, how much of the distillation pipeline (especially the offline cached-outputs mode) needs to be re-run, and what the day-to-day operational/on-call cost of running a live multi-teacher training system looks like.How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation — read next: extract-netflix-genpage-llm-homepage#g1 raises the identical missing on-call/operational-cost question for a different production ML-ranking system migration — read the two together as a cross-corpus pattern of infra pieces omitting operational cost.