RLHF / Post-Training Pipeline (SFT → Reward Model → RLHF)
The three-stage training pipeline (SFT on human demonstrations → reward model on preference comparisons → RL/PPO optimization against the reward model) that transforms a raw pretrained model into an instruction-following, safety-aligned assistant — originated in OpenAI's InstructGPT paper and still the conceptual foundation for post-training.
Also known as: RLHF, reinforcement learning from human feedback, supervised fine-tuning, SFT, reward model, reward modeling, preference data, preference model, PPO, proximal policy optimization, rollout, KL penalty, KL divergence penalty, policy model, InstructGPT pipeline, alignment tax
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| The Post-training Process OpenAI Used for ChatGPT Sharon Zhou · 2026-09-21 | rollout | A single prompt-response pair generated by the current policy during the RL loop; RL training operates on batches of rollouts. |
| The Post-training Process OpenAI Used for ChatGPT Sharon Zhou · 2026-09-21 | alignment tax | The tradeoff where optimizing for human preference (reward) via RL degrades the model's general pretraining-derived capabilities (knowledge recall, coherent long text) unless mitigated (e.g. by mixing in pretraining loss) — a form of catastrophic forgetting. |
Documents (1)
- The Post-training Process OpenAI Used for ChatGPT Sharon Zhou · 2026-09-21 Reader 🖥️
The behaviors that make frontier models like ChatGPT feel conversational, safe, tool-capable, and intelligent are not emergent side effects of pretraining scale but engineered outcomes of the post-training pipeline — supervised fine-tuning on human demonstrations, reward modeling on human preference comparisons, and reinforcement learning (PPO) against that reward model — and each stage of that pipeline produces a measurable, stacking improvement in response quality over the last.
Propositions touching this concept (11)
By publication month, oldest first (extraction month when no publication date is known).
2026-03
- theoretical interpretationCurrent RLHF objectives reward relevance/responsiveness to the immediate prompt in ways that produce models which mirror and elaborate user organizational structure rather than restructuring or simplifying it — creating a self-reinforcing cycle.
2026-04
- implicitThe training paradigm (RLHF and variants) is the only serious approach currently being pursued for alignment.
- explicitcorroboratedSimply telling the model to 'be brief' does not significantly reduce output tokens because RLHF-trained politeness is too deeply ingrained.
2026-07
- explicitdevelopedPost-training forces an early, locked-in architectural tradeoff between the flexible-but-complex RLHF family and the simpler, more scalable DPO family, and this choice must be settled before fine-tuning begins.
2026-09
- explicitThe classic ChatGPT/InstructGPT pipeline has three stages that build on each other: supervised fine-tuning (SFT) on human demonstrations, training a reward model on human preference comparisons, and reinforcement learning with human feedback (RLHF, via PPO) that optimizes the SFT model against the reward model.
- explicitHow a model is trained can matter as much as how large it is: human evaluators preferred a 1.3B-parameter RLHF model's outputs over a 175B-parameter SFT-only model's outputs most of the time.
- explicitTool use and function calling are taught almost entirely through supervised fine-tuning initially, with reinforcement learning subsequently refining correct versus incorrect/unnecessary invocations.
- implicitBecause the reward model is trained on one specific labeler population's preferences, RLHF optimizes the model toward that population's specific preferences rather than toward a population-neutral standard of 'good response' — evaluation by a different, held-out population shows the same preference gains but noticeably diminished.
- explicitJev is trained with a new method, Reinforcement Learning for Calibrated Decisions (RLCD), which optimizes for calibrated, epistemically honest probabilities on System One tasks, distinct from RLHF (human preference) and RLVR (verifiable rewards).
- explicitdevelopedThe author frames the founding motivation autobiographically: having helped build the RLHF-era methods behind ChatGPT at OpenAI, the author concluded that chat-optimized LLMs were missing something essential for automating real-world software decisions, motivating a four-year effort (two years in stealth) culminating in Jev.
- explicitPre-fetching and combining schedule and preference data into one tool call, rather than letting the agent search for each game itself, reduces the work and error surface of the agent's task.
Open gaps (10)
- incompleteThe is-ought argument assumes AI cannot develop functional analogs to emotion or sympathy. It treats the Humean framework as requiring literal biological evolution, ignoring the possibility that sufficiently complex information processing systems might develop emergent motivational states or that sympathy could be architecturally encoded.AI Alignment Is Impossible — read next: Affective computing, computational models of empathy, functionalist philosophy of mind (e.g., can functional states play the role of emotions?). Also work on reward modeling as a form of value encoding.
- incompleteGeneralization across model scale, architecture, and training recipe is largely untested in this summary post. Findings are drawn from specific Claude models (Sonnet 4.5, Opus 4.6) plus one independent replication on an open-weights model via Neuronpedia (credited to Neel Nanda's commentary) — leaving open whether J-space-like structure is a general property of transformer LLMs at scale or partly an artifact of Anthropic's own post-training recipe (e.g. Constitutional AI / RLHF choices), especially given the post's own finding that post-training substantially reshapes what enters the J-space.A global workspace in language models — read next: Read Neel Nanda's commentary (linked in the piece) for the specifics and limits of the open-weights replication.
- alternative explanationsThe report attributes all quality regression to thinking depth reduction but does not consider alternative causes: model weight updates, RLHF tuning changes, system prompt modifications, or context window handling changes that may have occurred in the same period.[MODEL] Claude Code is unusable for complex engineering tasks with the Feb updates #42796 — read next: Investigate Anthropic's model changelog for the February-March 2026 period to see what other changes coincided with the thinking reduction.
- incompleteThe paper does not investigate whether over-editing correlates with RLHF training that optimizes for helpfulness. Models may be trained to be maximally helpful, which in a coding context manifests as 'improving' code proactively. This would be a root-cause analysis the paper does not attempt.
- missing evidenceThe claim that models from different families have complementary blind spots is the core theoretical justification, but no analysis of what specific biases differ between Claude and GPT families is provided.GitHub Copilot CLI combines model families for a second opinion — read next: Look for research on training-data or RLHF-induced biases across model families; compare failure modes on shared benchmarks.
- uninformedThe claim that 'underlying algorithms haven't changed since the 1950s' is misleading. While perceptrons date to the 1950s, transformers (2017), RLHF, constitutional AI, chain-of-thought, and scaling laws represent genuine algorithmic innovations. The framing conflates 'neural networks exist since the 1950s' with 'nothing has changed.'Presentation: Leadership in AI-Assisted Engineering — read next: Vaswani et al. 'Attention Is All You Need' (2017); Kaplan et al. scaling laws papers; survey of post-transformer architectural innovations
- incompleteThe piece names that OpenAI 'contracted ~40 human labelers' and flags that model behavior is shaped by whichever population created the training/preference data, but never examines who those labelers were, how they were trained, compensated, or managed — a significant omission given the article's own admission that labeler population materially determines what the model optimizes for.The Post-training Process OpenAI Used for ChatGPT — read next: Reporting and research on RLHF/data-labeling labor conditions (e.g. contractor working conditions at outsourced annotation firms) and on labeler demographic effects on model behavior.
- RLHF interpretation as mechanism vs analogyThe Discussion proposes that prompt→response spillover reflects RLHF optimizing for relevance and helpfulness. This is plausible but mechanistic only by analogy — the paper does not actually instrument the model or correlate with known RLHF training details. Newer post-training paradigms (Constitutional AI, RLAIF, weighted preference learning, reasoning models) may produce different mirroring behaviors.Precision Proactivity: Measuring Cognitive Load in Real-World AI-Assisted Work — read next: Examine whether reasoning-trained models (o-series, Claude 3.7+ extended thinking, GPT-5 reasoning modes) exhibit different prompt-mirroring spillover under the same measurement framework. Stiennon et al. 2020 and Ouyang et al. 2022 (refs. 110-111) are the cited RLHF foundations.
- incompleteThe interview is a podcast transcript — the dialogue form allows Mounk to push back but not to synthesize. Several threads are opened and not closed: the precise mechanism of the alignment tax; whether Chinese political safety and alignment safety are actually converging or just using similar words; what 'frontier model' means precisely for open-source prohibition purposes.Sebastian Mallaby on AI Safety and the Race for Superintelligence — read next: Mallaby's book 'The Infinity Machine' on DeepMind presumably develops these arguments more carefully
- incompleteThe goblin incident is treated as a curiosity with alignment implications, but the piece doesn't explore the depth of those implications — what does it mean for safety training specifically when RL quirks cascade through model stacks?The Week AI Grew Up — read next: Anthropic's research on reward hacking; papers on RLHF failure modes and distributional shift