Purpose
This article explains the main post-training optimization methods used to shape LLM behavior after pre-training and supervised fine-tuning, including RLHF, verifier-based RL, DPO, ORPO, and broader search-plus-reward approaches.
Overview
After pre-training, a model has broad generative capability but is not necessarily optimized for the outputs humans prefer. After SFT, it may follow instructions better, but there is still often a gap between:
- syntactically valid answers and preferred answers
- plausible answers and verified answers
- fluent answers and safe / aligned / useful answers
Post-training methods address this gap by optimizing against some notion of preference or reward.
The core ingredients are usually:
- a policy model that generates outputs
- a signal that scores or ranks those outputs
- an optimization procedure that updates the policy toward higher-scoring behavior
Different methods vary mainly in how the signal is obtained and how the optimization is performed.
RLHF
RLHF stands for Reinforcement Learning from Human Feedback.
In the classical formulation, the workflow is:
- collect prompt-completion pairs
- obtain human preference comparisons between candidate completions
- train a reward model on those comparisons
- optimize the policy model to increase reward while staying near a reference policy
Reward model
The reward model takes a prompt and completion and predicts a scalar score representing human preference. Training data usually consists of ranked or pairwise comparisons:
- completion A preferred over completion B
The reward model is therefore a learned proxy for human preference, not a ground-truth correctness function.
Policy optimization
Historically, PPO was the dominant algorithm used to optimize the policy. The objective was not “maximize reward at any cost”: a KL penalty to a reference policy was typically added to prevent the model from drifting too far from the SFT distribution.
This matters because unconstrained reward maximization can lead to reward hacking, verbosity inflation, or degenerate policy drift.
PPO in RLHF
PPO is important historically because it operationalized RLHF in many early large-scale systems.
The practical objective typically includes:
- expected reward from the reward model
- KL penalty versus a frozen reference model
- value function estimation for advantage computation
Operationally, PPO-based RLHF requires:
- rollouts from the current policy
- reward computation
- advantage estimation
- policy and value updates
- careful stability tuning
This makes PPO pipelines more complex and expensive than pure supervised objectives.
Common RLHF pathologies
- reward overoptimization
- verbosity as a proxy for preference
- instability under weak KL control
- exploitation of reward model blind spots
- mismatch between preference labels and real task success
RLVR / verifier-based RL
RLHF is one instance of a broader post-training pattern:
generate candidate outputs
score those outputs using some signal
optimize the policy (LLM weights) toward higher-scoring behaviour
The important variable is therefore not just the optimization algorithm (such as PPO), but the source of the reward signal itself. In classical RLHF, the reward signal comes from human preference judgements. A reward model is trained to predict which completions humans tend to prefer. However, not all tasks are best evaluated through human preference.
For some domains, correctness can be evaluated directly by an external system rather than inferred from human preference judgments. In these cases, the reward signal can come from a verifier instead of a learned human-preference model.
This leads to verifier-based RL, where outputs are rewarded according to whether they satisfy externally checkable criteria rather than whether humans merely prefer them.
Examples:
- unit tests for code
- theorem provers
- symbolic checkers
- exact-match evaluators for math or structured tasks
- rule-based validators for formal outputs
This is powerful because a deterministic or high-precision verifier often provides a sharper signal than noisy human preference labels.
Why verifier-based training matters
For domains where correctness is machine-checkable, verifier reward can scale much more cheaply than human comparison labeling. It is especially attractive for:
- code generation
- formal reasoning
- constrained structured outputs
However, verifier quality still matters. A narrow verifier can induce optimization toward passing the checker rather than solving the underlying task robustly.
DPO
DPO (Direct Preference Optimization) removes the need to train an explicit reward model and run a separate RL loop.
Instead, it learns directly from preference pairs using a supervised-style objective derived from the relationship between rewards and optimal policies under KL-constrained RL.
At a high level, DPO encourages the policy to increase probability on preferred completions relative to rejected completions, compared with a reference model.
Why DPO became important
DPO is attractive because it is simpler than full RLHF:
- no separate reward-model training stage
- no rollout/value/PPO machinery
- easier optimization and implementation
It often achieves strong results in practice for instruction-following and preference alignment tasks.
Important nuance
DPO is still not “free.” It depends on:
- preference data quality
- choice of reference model
- beta / temperature-like tradeoff parameter
- data mixture with SFT-like examples
Poor preference pairs still yield poor policy shaping.
ORPO
ORPO (Odds Ratio Preference Optimization) combines language-model-style optimization and preference learning into a single objective.
Instead of treating SFT and preference optimization as strictly separate phases, ORPO integrates them in one loss formulation. The intent is to reduce pipeline complexity and improve efficiency.
Why this is useful
In practice, many post-training pipelines have multiple stages:
- SFT
- preference tuning
- optional RL or verifier tuning
A unified or partially unified objective can reduce handoff complexity between stages and may be easier to operationalize for some training workflows.
The important conceptual point is that ORPO sits between:
- plain supervised imitation
- pairwise preference shaping
Preference optimization is not the same as correctness optimization
A critical distinction:
- preference models learn what raters tend to like
- verifiers check whether an answer satisfies a formal criterion
- task reward may be neither of the above exactly
This distinction explains why a model can be:
- pleasant but incorrect
- correct but stylistically poor
- safe but unhelpful
- high-scoring under one reward but bad under another
Post-training quality depends on matching the optimization signal to the real deployment objective.
RL and search for reasoning models
For reasoning-heavy systems, the post-training story increasingly includes not just policy learning, but search.
A common pattern is:
- generate multiple candidate trajectories
- score them with reward, verifier, or outcome
- update the policy to reproduce stronger trajectories
This connects LLM training with broader RL ideas:
- policy improvement
- exploration vs exploitation
- credit assignment
- search-guided target generation
Why search matters
Hard reasoning tasks often have sparse or delayed reward. A model may need search to discover successful trajectories before gradient-based learning can reinforce them.
This is why methods inspired by AlphaZero-like loops, rejection sampling, best-of-N selection, or MCTS-style reasoning have become important in advanced reasoning pipelines.
Rejection sampling and offline preference shaping
Not all alignment or preference optimization requires online RL. A practical middle ground is:
- sample multiple candidates from a policy
- filter using a reward model, verifier, or heuristics
- fine-tune on the kept candidates
This is simpler than PPO and often works surprisingly well. It is especially common when:
- inference is cheap enough to generate candidates offline
- a verifier exists
- the training team wants a simpler pipeline than online RL
Why this matters in practice
These methods explain why two models with similar base pre-training can behave very differently after post-training.
They also explain why “alignment” is not one thing. Different pipelines optimize for different targets:
- human preference
- safe style
- verifier-checked correctness
- instruction compliance
- long-horizon reasoning success
Operationally, these distinctions affect:
- data collection design
- training complexity
- evaluation methodology
- failure modes in deployment
Key takeaways
- RLHF uses human preference comparisons to train a reward proxy and optimize policy behavior.
- PPO-based RLHF is historically important but operationally complex.
- Verifier-based RL is especially effective when correctness can be checked automatically.
- DPO simplifies preference optimization by removing the explicit reward-model-plus-RL loop.
- ORPO integrates instruction-style learning and preference shaping in a single objective.
- For reasoning systems, policy quality often depends on the interaction between reward, verifier quality, and search.
Comments
0 comments
Article is closed for comments.