The Post-Training Stack: SFT, Reward Models, and RL

This is Part 1 of the Post-Training & Alignment series.

Pre-training ends when the model has seen enough tokens to predict the next one with statistical accuracy. Post-training begins when we need the model to do something specific: follow instructions, refuse harmful requests, or adhere to a tone. This transition is rarely a single step. It is a pipeline comprising three distinct mechanisms: Supervised Fine-Tuning (SFT), Reward Modeling (RM), and Reinforcement Learning (RL). Each stage addresses a specific limitation of the previous one. Understanding this stack is critical because treating these stages as interchangeable or conflating them leads to models that are either unhelpful or unstable.

Supervised Fine-Tuning (SFT): The Anchor

The base language model is a next-token predictor trained on massive corpora of web text. It does not inherently understand the concept of a "query" and an "answer." If you ask a base model, "What is the capital of France?", it might complete the sentence with "Paris is the largest city in France..." rather than simply stating "Paris."

SFT solves this by performing behavior cloning on a curated dataset of instruction-response pairs $(x, y)$. Here, $x$ is the prompt (including system instructions) and $y$ is the desired output. The objective is standard cross-entropy loss, identical to pre-training but applied only to the response tokens.

$\mathcal{L}_{SFT} = -\sum_{i=1}^{n} \log P_{\theta}(y_i | x, y_{<i})$

This stage does not involve any notion of "good" or "bad." It only involves "correct format." The mechanism here is to shift the model’s probability mass from the broad, unstructured distribution of web text to a narrower, more structured distribution of helpful interactions. Without SFT, the subsequent RL stage would have nothing stable to optimize. If you attempt to apply RL directly to a base model, the policy is too volatile, and the reward signal is too noisy to guide meaningful improvement.

Reward Modeling (RM): The Proxy for Human Preference

Once the model can generate formatted responses, we need a way to rank them. Humans cannot manually grade every possible output, so we train a separate model, the Reward Model, to approximate human judgment.

The RM is typically initialized as a fine-tuned version of the same architecture used for SFT. It takes a prompt $x$ and a response $y$ and outputs a scalar score $r(x, y)$. The training data consists of human-labeled pairwise comparisons: for a given prompt $x$, humans label which of two responses, $y_w$ (winning) or $y_l$ (losing), is better.

The RM is trained using a contrastive loss, often based on the Bradley-Terry model. The goal is to maximize the margin between the score of the preferred response and the non-preferred response:

$\mathcal{L}_{RM} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} [\log \sigma(r(x, y_w) - r(x, y_l))]$

Here, $\sigma$ is the sigmoid function. This mechanism teaches the RM to output higher values for responses that align with human preferences (helpfulness, honesty, harmlessness) and lower values for others. Crucially, the RM is not the policy. It is a static evaluator. It does not generate text; it scores text. This separation of concerns is vital. If the generator and the evaluator are the same model during training, the model can "cheat" by generating text that tricks its own evaluation head, a phenomenon known as reward hacking.

Reinforcement Learning: Optimizing for the Long Term

With an SFT model as the initial policy $\pi_{SFT}$ and an RM as the reward function $r(x, y)$, we enter the RL stage. The most common algorithm used here is Proximal Policy Optimization (PPO).

In this setup, the LLM acts as the agent. It generates a response $y$ for a prompt $x$. The RM evaluates this response, producing a reward $r(x, y)$. The RL algorithm updates the LLM's parameters to maximize this expected reward. However, a naive maximization of the RM score leads to degenerate solutions. The model might discover that generating nonsensical but highly confident text, or repeating certain phrases, yields high scores from the RM, even if the content is meaningless. This is the "reward hacking" problem again, but at the policy level.

To prevent this, the RL objective includes a KL-divergence penalty term that measures the distance between the current policy $\pi_\theta$ and the reference SFT policy $\pi_{SFT}$:

$\max_\theta \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)} [r(x, y) - \beta D_{KL}(\pi_\theta(y|x) || \pi_{SFT}(y|x))]$

The parameter $\beta$ controls the strength of this penalty. The mechanism here is crucial: it ensures that the RL policy stays close to the SFT distribution. The SFT policy provides the "grammar" and "style" of the language, while the RM provides the "preference" signal. The KL penalty prevents the model from drifting too far into the mode collapse regions where the RM is overconfident but the text is incoherent.

Why the Stack Matters

The separation of SFT, RM, and RL is not arbitrary. It reflects the different types of errors each stage corrects, and critically, it prevents error propagation between stages. If these components were merged or removed, the model would suffer from compounding instability.

1. SFT corrects for lack of instruction following. It teaches the model how to speak. By establishing a stable distribution of coherent, formatted responses, it prevents the RL phase from starting in a region of high entropy where optimization gradients are meaningless. Without this anchor, the RL policy would drift into incoherence immediately.
2. RM corrects for lack of preference understanding. It teaches the model what is better. By acting as a distinct proxy for human values, it isolates the evaluation signal from generation. This separation ensures that the model doesn't conflate fluency with correctness, a common failure mode in single-stage optimization.
3. RL corrects for suboptimal policy selection. It teaches the model to choose the better path in its generation process. The KL penalty, rooted in the SFT reference, ensures that improvements in reward do not come at the cost of linguistic coherence.

Removing any component breaks the alignment. Without SFT, RL has no stable starting point. Without RM, RL has no signal. Without RL, the model remains stuck in the average of the SFT data, unable to explore beyond the immediate next-token optimization of supervised learning.

Recent research, such as Direct Preference Optimization (DPO), attempts to simplify this stack by removing the explicit RM and RL steps, optimizing the policy directly against human preferences using a pairwise loss. However, DPO is effectively a re-parameterization of the same underlying trade-off between reward maximization and KL regularization. The conceptual dependency on a preference signal and a reference policy remains. Understanding the full stack provides the necessary intuition for debugging failures in any variant of post-training, whether it uses PPO, DPO, or future algorithms.

Conclusion

The post-training stack is a layered solution to the misalignment between pre-trained weights and human intent. SFT provides the structural foundation, RM provides the directional signal, and RL optimizes the policy within safe boundaries. While newer methods like DPO offer streamlined alternatives, they still rely on the fundamental distinctions established by this three-stage pipeline. Understanding these components is essential for any engineer working on aligned language models.

Audit your current post-training pipeline: Are you conflating SFT formatting with RM preference signals? Check your loss curves for reward hacking.

Common Pitfalls

1. Using a base model for RL without SFT: Attempting to optimize a raw base model with RL leads to immediate policy collapse. The model lacks the basic instruction-following capabilities required to interpret prompts, resulting in nonsensical outputs that cannot be meaningfully scored by an RM.
2. Training RM on non-comparable pairs: Reward models trained on pairs that are not truly comparable (e.g., different topics or vastly different lengths without normalization) learn noisy preferences. This leads to an RM that rewards superficial features like length or keyword density rather than genuine helpfulness or safety.
3. Ignoring KL penalty in PPO: Setting the KL penalty coefficient ($\beta$) too low allows the policy to diverge significantly from the SFT reference. This results in "reward hacking," where the model exploits quirks in the RM to achieve high scores while producing gibberish or repetitive text.

Practical Takeaways

1. SFT is for format, RM is for preference: Treat SFT as the step where you teach the model the structure of a good answer (concise, polite, structured). Treat RM as the step where you teach the model which of two valid answers is actually more useful or safe.
2. RM is a static evaluator, not a generator: Never confuse the Reward Model with the Policy Model. The RM scores; the Policy generates. Keeping these roles distinct prevents the model from gaming its own evaluation metric.
3. KL penalty prevents mode collapse: The KL divergence term is not just a regularizer; it is the safety rail that keeps the model within the distribution of coherent language defined by SFT. Always monitor the KL divergence during RL training to ensure the model hasn't drifted too far.

FAQ

1. What is the difference between SFT and RM? SFT (Supervised Fine-Tuning) trains the model to generate specific types of responses by mimicking human examples. It focuses on formatting and instruction following. RM (Reward Modeling) trains a separate model to score responses based on human preferences. It focuses on ranking and preference alignment. SFT teaches the model how to speak; RM teaches it what sounds better.

2. Why is KL divergence used in RL? KL divergence is used as a penalty term in the RL objective to measure the distance between the current policy and the reference SFT policy. It prevents the model from deviating too far from the coherent, formatted language learned during SFT. Without this penalty, the RL process might optimize for high reward scores by generating incoherent or repetitive text that happens to trick the Reward Model.

3. Can DPO replace the entire stack? DPO (Direct Preference Optimization) simplifies the stack by removing the explicit Reward Model and RL steps, optimizing the policy directly using pairwise preferences. However, it does not replace the conceptual need for a preference signal and a reference policy. DPO still relies on the SFT model as a reference to prevent mode collapse, similar to the KL penalty in PPO. It streamlines the engineering but retains the underlying dependencies of the full stack.