How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation

Claudio Masolo · rss · 2026-09-11 source ↗ Reader 🖥️

AI Unit Economics Compute & Energy Infrastructure Feedback Loops Inference Optimization Knowledge Distillation (Teacher-Student Compression)

LinkedIn made multi-teacher knowledge distillation practical for its production job-search ranking system not through a distillation-algorithm breakthrough but through training-infrastructure engineering — a custom SGLang-based framework that serves teacher models directly inside the training loop (an online mode for fast iteration, an offline precomputed-and-cached mode once teachers stabilize) plus a stack of largely independent training-speed optimizations — collectively yielding a roughly eightfold training speedup that lets a compact 0.6B student model approach cross-encoder-quality ranking while avoiding the cost and latency of calling a frontier LLM on every search request.

practical · trade/technical-news digest of a vendor engineering blog post — This is an InfoQ-authored summary of LinkedIn's own engineering blog post (plus a second, linked LinkedIn research post on search-stack ranking), not a first-person technical writeup or a peer-reviewed paper. All numbers are vendor-reported and single-sourced through InfoQ's relay. Extraction is deliberately kept on the training-infrastructure/engineering angle (cost avoidance, latency, iteration speed, systems design) rather than the ranking-model architecture itself, which is peripheral to the reader's core interests.

Structure

Part	Role
Framing	Asserts up front that the article's real subject is systems/infrastructure engineering, not the distillation technique itself
Problem statement	States the bottleneck: querying teacher models during training is slow, and a ranking system handling hundreds of thousands of QPS can't afford that latency; generalizes this to any team moving from keyword search to LLM-supervised unified ranking
System architecture	Describes the custom SGLang framework (tensor/data-parallel teacher serving, async client), Online Multi-teacher Distillation (live queries, 3x speedup via multi-node local teacher replicas), and Offline Multi-teacher Distillation (precomputed teacher outputs cached to HDFS/NFS)
Additional training optimizations	Lists LiGer (2x batch size), multi-node training (3.5x), FSDP2 (+20%), H200 clusters (+30%), and the rejected FP8 mixed-precision experiment, attributing the roughly 8x headline speedup to their combination
Modeling and inference results	Cites the 0.6B student model's NDCG@10 improvement (0.7583 to 0.9432) from a separate, linked LinkedIn research post, plus inference-side throughput gains (290 to 2,000+ items/sec/GPU) from structured pruning and context compression
Production deployment claim	States the system is live for US job search on SGLang (not a proprietary stack) and generalizes the online-then-offline teacher-serving pattern into transferable advice for other teams building LLM-supervised ranking
Industry context	Situates the work against Pinterest's related training-scalability effort and a survey of frontier-model multi-teacher distillation (10+ teachers), to scope industrial ranking distillation as a simpler regime than frontier-model distillation

Key terms

Online Multi-teacher Distillation
LinkedIn's name for querying multiple teacher models live, inside the training loop, via an asynchronous client that feeds their outputs directly into the distillation losses each step — scaled across nodes with local teacher replicas for a 3x speedup.
Offline Multi-teacher Distillation
The complementary mode: precomputing all teacher outputs ahead of time and storing them on HDFS or NFS, so training reads cached labels instead of querying teachers live — used once teacher choice and query volume stabilize, to eliminate serving overhead entirely.
Multi-teacher distillation framework
The whole custom SGLang-based system — model loading, tensor/data-parallel serving, the async client, and the online/offline switch — described as a single named artifact, folding serving infrastructure into what 'distillation' means in this context.
8x faster
A rounded, composite figure for the aggregate training/iteration speedup attributed to stacking five distinct interventions (the multi-node online-distillation architecture, LiGer, multi-node data-parallel training, FSDP2, H200 clusters) — not the measured result of any single technique.

Propositions

explicitdevelopedLinkedIn built a custom multi-teacher distillation framework on SGLang that serves teacher models directly inside the training loop, compressing knowledge from large teacher models into a compact 0.6B-parameter ranking model for AI-powered job search.conf: high
extends ←Transcript for Cursor Team: Future of Programming with AI | Lex Fridman Podcast #447 · p23 — The Cursor team's RL-based approach to getting frontier-level perceived quality out of a small, fast model (via reward signal from preferred completions) is an earlier, lighter-weight instance of the same general move the later piece formalizes as structured multi-teacher knowledge distillation.
explicitThe main contribution of this work is not the distillation technique itself but the systems engineering that makes distillation fast enough for rapid iteration.conf: high
explicitServing teacher models for supervision during training becomes a bottleneck for a ranking system handling hundreds of thousands of queries per second, which motivates splitting the approach into online (live-query) and offline (precomputed/cached) multi-teacher distillation modes.conf: high
explicitScaling Online Multi-teacher Distillation across multiple nodes with local teacher replicas accelerated the distillation process by 3x while keeping latency low enough for quick iterations.conf: medium
explicitOffline Multi-teacher Distillation — precomputing teacher outputs and storing them on HDFS or NFS instead of querying teachers live — reduces serving overhead and prevents repeated computation.conf: high
explicitStacking multiple training-level optimizations — LiGer (2x larger batch sizes), multi-node training (up to 3.5x), FSDP2 (a further 20%), and H200 multi-node clusters (up to 30% more) — is what accounts for the roughly eightfold training speedup referenced in the title.conf: medium
extendsGuest post: AI Inference Is Breaking Unit Economics · p4 — Extends the general 'stacking optimization techniques yields multiplicative gains' pattern (there: Yandex's 5.8x via quantization + speculative decoding + KV cache reuse + parallelization, for inference) to a training-speed context — while also complicating it, since the stated aggregate figure here doesn't cleanly follow from the individual multipliers cited.
explicitLinkedIn evaluated FP8 mixed precision training and found no benefit for models under 8B parameters, because casting overhead outweighs any compute savings at that scale — a negative interaction in an otherwise cumulative, positive optimization-stacking story.conf: high
supportsGuest post: AI Inference Is Breaking Unit Economics · p7 — Both are instances of the same pattern — a precision/quantization-style optimization (FP8 here, aggressive quantization there) that looks good in isolation but has a negative interaction cost (casting overhead; noisier speculative-decoding verification) that erodes or eliminates the gain under certain conditions.
explicitThe 0.6B student model, distilled from an 8B relevance oracle and a 1.7B engagement teacher, raised NDCG@10 for job search results by 24.48% (from 0.7583 to 0.9432).conf: medium
explicitSeparate inference-side optimizations — structured pruning and context compression — increased ranking throughput from about 290 to over 2,000 items per second per GPU (roughly 7x).conf: medium
explicitThe online-then-offline teacher-serving split generalizes as a lifecycle strategy for any team building LLM-supervised ranking systems: query teachers live while teacher choice is still unstable in early stages, then switch to offline caching once teachers stabilize and query volume increases.conf: high
implicitBuilding the training/serving framework on the open-source SGLang engine (rather than a proprietary stack LinkedIn had already invested in for prior ranking workloads) was a deliberate strategic choice to reuse existing infrastructure investment and avoid vendor lock-in.conf: high
supportsGuest post: AI Inference Is Breaking Unit Economics · p5 — Corroborates the claim that most optimization gains are achievable by engineering teams using open-source tools without a dedicated research lab — LinkedIn built its custom framework on open-source SGLang rather than a proprietary stack.
explicitPrecision-reduction techniques like FP8 have a model-size threshold below which their overhead outweighs their savings, so teams should not assume lower-precision-by-default is universally beneficial.conf: high

Arguments

The systems/infrastructure work — the custom SGLang-based multi-teacher serving framework and its online/offline modes — rather than the distillation technique itself, is what made multi-teacher distillation practical for LinkedIn's production-scale ranking system. strength: medium

Explicit premises:

Serving teacher models during training is a bottleneck at hundreds-of-thousands-of-QPS scale.
A custom SGLang framework was built to load and serve teachers of different sizes with tensor-parallel and data-parallel setups plus an asynchronous client.
Online mode gave a 3x speedup with low latency by scaling across nodes with local teacher replicas.
Offline mode further cut serving overhead by precomputing and caching teacher outputs.

Implicit premises:

The distillation technique's value (the ranking-quality gain measured separately as +24.48% NDCG@10) was already available or achievable before this infrastructure existed — the constraint being removed is an engineering/systems bottleneck, not a lack of modeling technique.
Iteration speed — how fast a team can retrain, re-evaluate, and adjust teacher choice — is the scarce production-ML resource here, not raw training compute or elegance of the distillation loss function.
The 'roughly 8x' headline number is primarily valuable as a case for infrastructure investment, not as an inherent property of the distillation method.

Stacking independently-adopted training optimizations (the multi-node online-distillation architecture, LiGer, multi-node data-parallel training, FSDP2, H200 clusters) compounds into a large aggregate speedup, without needing FP8 for models under 8B parameters. strength: low

Explicit premises:

LiGer gives 2x larger batch sizes / reduced memory use.
Multi-node training gives up to 3.5x additional speedup.
FSDP2 gives a further 20% gain.
H200 multi-node clusters give up to 30% more on top of that.
FP8 mixed precision was tested and rejected for models under 8B parameters due to casting overhead outweighing compute savings.

Implicit premises:

These optimizations are sufficiently independent/orthogonal that their gains can be adopted incrementally rather than requiring a monolithic redesign.
Precision-reduction techniques have a model-size threshold below which overhead exceeds savings, implying the compute-bound/memory-bound regime shifts with model scale.
The stated ~8x figure is a rounded, holistic characterization of the whole pipeline rather than the literal arithmetic product of the individually cited multipliers.

Narratives

A companion account from Pinterest engineering (June 2026), 'Achieving Near-Linear Training Scalability for Pinterest's Foundation Models,' describing Pinterest scaling its training framework and migrating to Distributed Checkpoint to make multi-node teacher training possible, then distilling larger teachers into efficient student models for Homefeed and Related Pins ranking — cutting experimentation cycles from weeks to a fraction of that.
→ Implicitly supports the claim that training-infrastructure investment for distillation iteration speed is a general industry pattern rather than a LinkedIn-specific quirk, while also implicitly distinguishing LinkedIn's specific contribution (live/online teacher-querying during training) from Pinterest's (checkpointing/scaling the training framework itself).
A reference to the survey 'Distillation in 2026 (so far): which frontier models use it and how,' noting that frontier models like NVIDIA's Nemotron 3 Ultra, MiMo-V2-Flash, and DeepSeek-V4 use ten or more specialized teachers for dense, token-level supervision.
→ Functions as a scoping/contrast narrative: implicitly argues that industrial ranking distillation (LinkedIn's, Pinterest's) is a simpler regime — fewer, task-specific teachers — than frontier-model distillation, justifying LinkedIn's two-teacher setup as proportionate to the problem rather than under-engineered.

Questions addressed

✅ How can a team supervise a small, cheap-to-serve ranking model with large teacher LLMs without teacher-serving latency becoming a bottleneck to training iteration?
Build a framework that can serve teachers directly inside the training loop (online mode) when teacher choice is still unstable, and switch to precomputed/cached teacher outputs (offline mode) once teachers and query volume stabilize.
⭕ How much of the total training speedup comes from which specific optimization, and how do the cited multipliers combine into the stated 'roughly 8x' headline?
✅ Does distilling from multiple teacher models (a relevance oracle plus an engagement teacher) actually improve downstream ranking quality over a single-teacher or non-distilled baseline?
Cited NDCG@10 improvement of 24.48% (0.7583 → 0.9432) for the 0.6B student model — though the baseline being compared against is not specified.
✅ Is FP8 mixed precision worth adopting by default when training sub-frontier-scale models?
No — LinkedIn found casting overhead outweighs compute savings for models under 8B parameters.

Gaps

illogicalThe 'roughly eightfold' training speedup headline cannot be reconstructed from the individually cited multipliers (3x from the multi-node online-distillation architecture, 2x batch size from LiGer, up to 3.5x from multi-node training, +20% from FSDP2, up to 30% from H200 clusters) — naively multiplying them yields a number well above 8x, and the piece never shows which multipliers overlap, are additive rather than multiplicative, or apply to different, non-cumulative pipeline stages.
Read next: Read LinkedIn's original engineering blog post directly for the actual before/after wall-clock benchmark and the ablation methodology isolating each optimization's contribution.
incompleteNo cost accounting is given anywhere — GPU-hours, H200 cluster cost, or engineering time to build the custom SGLang framework — despite the piece's own framing that the system's purpose is to avoid the high cost of frontier-LLM inference per request. A reader cannot tell whether the infrastructure investment paid for itself relative to the cost it was built to avoid.
Read next: extract-ai-inference-unit-economics for a total-cost-of-ownership framing that prices infrastructure investment against inference-cost savings.
incompleteThe NDCG@10 improvement (0.7583 → 0.9432) and the throughput improvement (290 → 2,000+ items/sec/GPU) are attributed to a separate, only loosely specified piece of LinkedIn research (structured pruning and context compression), and it is unclear whether both figures come from the same deployed 0.6B student model, the same evaluation set, or the same production conditions.
Read next: The linked LinkedIn 'Reimagining LinkedIn's Search Stack' post for the full evaluation methodology and whether the quality and throughput figures share the same underlying system.
uninformedThe piece doesn't engage with why LinkedIn needed a fully custom SGLang framework rather than adapting an existing distillation-serving toolkit — no comparison of build-vs-adapt cost or capability gap is offered, so a reader can't judge whether 'custom framework' reflects a genuine gap in existing tools or an organizational preference for owning the infrastructure.
Read next: extract-netflix-in-house-llm-serving-platform-triton-vllm#g2 raises the identical missing-comparison problem for a different production LLM-serving stack decision; industry surveys of vLLM, SGLang, TensorRT-LLM, and DeepSpeed-Inference for feature-parity assessment.
incompleteNo discussion of model/quality drift or ongoing maintenance cost: as the teacher models themselves get updated or replaced, how much of the distillation pipeline (especially the offline cached-outputs mode) needs to be re-run, and what the day-to-day operational/on-call cost of running a live multi-teacher training system looks like.
Read next: extract-netflix-genpage-llm-homepage#g1 raises the identical missing on-call/operational-cost question for a different production ML-ranking system migration — read the two together as a cross-corpus pattern of infra pieces omitting operational cost.

Closes gaps elsewhere

Guest post: AI Inference Is Breaking Unit Economics · g4 — Provides a concrete example of 'another negative interaction in the optimization stack': FP8 mixed precision was tested and rejected for training models under 8B parameters because casting overhead outweighs compute savings at that scale — the same 'stacking is not uniformly positive' pattern that gap asked about, though found on the training side rather than the inference side that gap was specifically scoped to.
Guest post: AI Inference Is Breaking Unit Economics · g5 — Directly fills the noted omission of model distillation as an inference-cost-reduction strategy: this is exactly a large-model-to-small-model distillation case, with LinkedIn's own quantified quality (NDCG@10 +24.48%) and throughput (~7x, 290→2,000+ items/sec/GPU) payoffs from doing it in production.
How Netflix Built GenPage: a Single GenAI Model to Build Personalized Homepages · g1 — Supplies exactly the training-infrastructure-cost detail that gap flagged as missing from the Netflix piece — a concrete architecture (custom SGLang framework, tensor/data-parallel teacher serving, online/offline modes) and quantified speedup contributions (3x, 2x, 3.5x, +20%, +30%) for migrating a ranking system to LLM-based supervision. It does not, however, address the on-call/debugging-complexity half of that gap.