The Post-training Process OpenAI Used for ChatGPT

Sharon Zhou · rss · 2026-09-21 Reader 🖥️

AI Alignment & Safety AI Productization Pipeline Extended Thinking / Reasoning MCP (Model Context Protocol)RLHF / Post-Training Pipeline (SFT → Reward Model → RLHF)

The behaviors that make frontier models like ChatGPT feel conversational, safe, tool-capable, and intelligent are not emergent side effects of pretraining scale but engineered outcomes of the post-training pipeline — supervised fine-tuning on human demonstrations, reward modeling on human preference comparisons, and reinforcement learning (PPO) against that reward model — and each stage of that pipeline produces a measurable, stacking improvement in response quality over the last.

theoretical · expository — Primarily explains and defends how a technical process works (post-training turns a raw pretrained model into ChatGPT), grounded in the InstructGPT paper's methodology and results, rather than arguing the reader should take a specific action. There is a thin practical undercurrent aimed at engineers ('getting it right takes significant engineering effort'), but the dominant mode is exposition of mechanism plus evidentiary argument for why the mechanism works.

Structure

Part	Role
Opening: the golden-retriever prompt before/after ChatGPT's November 2022 launch	context
Behavior survey: conversational/helpful, safety/alignment, tool use, reasoning	evidence — enumerates the specific capabilities post-training is claimed to produce
The classic ChatGPT/InstructGPT pipeline (three stages: SFT, reward modeling, RLHF)	mechanism — the detailed causal account of how the claimed behaviors are actually produced
Stage-by-stage deep dive: SFT data collection, reward-model architecture and training, the 9-step PPO/RLHF loop, KL-penalty and reward-hacking discussion	evidence — technical detail intended to make the mechanism credible and inspectable, not just asserted
'Getting it right' closing section: human-eval results (RLHF > SFT > base GPT-3; a 1.3B RLHF model beating a 175B SFT model) and the held-out-labeler caveat	conclusion — the empirical payoff that is meant to validate the whole pipeline, with a self-qualifying caveat about whose preferences were optimized

Key terms

post-training
The staged process applied after pretraining — SFT, reward modeling, RLHF, and by extension reasoning/tool-use training — that converts a raw next-token predictor into a usable assistant.
rollout RLHF / Post-Training Pipeline (SFT → Reward Model → RLHF)
A single prompt-response pair generated by the current policy during the RL loop; RL training operates on batches of rollouts.
reward hacking
The failure mode where an RL policy, if unconstrained, finds degenerate outputs that exploit quirks in the reward model to score highly without being good responses.
alignment tax RLHF / Post-Training Pipeline (SFT → Reward Model → RLHF)
The tradeoff where optimizing for human preference (reward) via RL degrades the model's general pretraining-derived capabilities (knowledge recall, coherent long text) unless mitigated (e.g. by mixing in pretraining loss) — a form of catastrophic forgetting.
preference data / preference model
Human rankings of multiple model responses to the same prompt, converted to pairwise comparisons; the reward model trained on this data is interchangeably called a 'preference model.'

Propositions

explicitPost-training, not raw model scale, is what converts a pretrained model into a usable conversational assistant.conf: high
explicitThe classic ChatGPT/InstructGPT pipeline has three stages that build on each other: supervised fine-tuning (SFT) on human demonstrations, training a reward model on human preference comparisons, and reinforcement learning with human feedback (RLHF, via PPO) that optimizes the SFT model against the reward model.conf: high
extendsPresentation: Engineering AI for Creativity and Curiosity on Mobile · p2 — Supplies the concrete SFT -> reward-model -> PPO mechanics behind the 'RLHF family' alternative that this proposition frames as one of two locked-in post-training tradeoffs (vs. DPO).
extendsIntroducing System One Models & Jev · p14 — Gives the actual technical account of the 'RLHF-era methods behind ChatGPT' that Jev's founder is credited with having helped build at OpenAI.
explicitSafety and helpfulness are fundamentally in tension, and where to draw the line between them is a human, organizational decision made within labs, not a technical given.conf: high
explicitThe raw capacity to reason is latent in pretrained models (from training data containing proofs, logic, and commented code) but pretrained models default to pattern-matching rather than deliberate step-by-step reasoning unless post-trained to reason.conf: high
explicitHow a model is trained can matter as much as how large it is: human evaluators preferred a 1.3B-parameter RLHF model's outputs over a 175B-parameter SFT-only model's outputs most of the time.conf: high
explicitPreventing reward hacking requires an explicit KL-divergence penalty that tethers the RL policy to the original SFT reference model, trading off reward maximization against distributional drift from that reference.conf: high
supportsCoding Models Are Doing Too Much · p13 — OpenAI's practice of mixing pretraining data into the SFT and RL objectives to prevent catastrophic forgetting is a concrete real-world instance consistent with this proposition's general distributional-distance account of forgetting.
supportsI benchmarked CC's caveman plugin against 'be brief'. The token math is wild. · p2 — The KL penalty's job is precisely to keep the RL policy close to the SFT-shaped reference model, which is the mechanism that would make RLHF-trained stylistic habits like excessive politeness resistant to being prompted away.
explicitTool use and function calling are taught almost entirely through supervised fine-tuning initially, with reinforcement learning subsequently refining correct versus incorrect/unnecessary invocations.conf: high
implicitBecause the reward model is trained on one specific labeler population's preferences, RLHF optimizes the model toward that population's specific preferences rather than toward a population-neutral standard of 'good response' — evaluation by a different, held-out population shows the same preference gains but noticeably diminished.conf: medium

Arguments

The three-stage RLHF pipeline reliably improves response quality at each successive stage, and the improvement comes from training method, not just scale. strength: medium

Explicit premises:

Human evaluators preferred RLHF outputs over SFT outputs, and SFT outputs over base GPT-3 outputs, in head-to-head comparison.
A 1.3B RLHF model was preferred over a 175B SFT-only model most of the time.
The RLHF model was better at following explicit constraints, less likely to produce harmful outputs, and hallucinated less than the SFT model.

Implicit premises:

Human labeler pairwise preference judgments are a valid, sufficiently general proxy for the qualities readers actually want from an assistant.
A reward model trained on one labeler population's preference data generalizes adequately to other users, not just to the population that produced its training data.

Inferences:

If preference gains measurably shrink under a held-out labeler population (as the article's own caveat shows), then reported RLHF 'win rate' style benchmarks likely systematically overstate real-world preference gains for user populations that differ from the original labeling pool — a caveat the article notes locally but does not generalize to the field's benchmarking practices.

A KL-divergence penalty tying the RL policy to the original SFT model is what keeps reward maximization from degenerating into reward hacking. strength: medium