GRPO and the Rise of Verifiable Rewards

Group Relative Policy Optimization (GRPO) is a significant advancement in aligning large language models (LLMs) for reasoning tasks. Part 4 of the Post-Training & Alignment series, this article examines how GRPO, a specific variant of RLVR introduced in the DeepSeekMath technical report, leverages verifiable rewards to enhance reasoning capabilities without relying on complex critic models. By replacing the traditional value estimator with a group-relative baseline, GRPO mitigates one source of instability (critic error) but introduces others (baseline variance). This approach reduces resource overhead while requiring careful tuning to prevent mode collapse, offering a stable, efficient path to improved model performance.

The Critic Bottleneck in Traditional RLVR

To understand the significance of GRPO, one must first understand the mechanics of Proximal Policy Optimization (PPO) in the context of LLMs. In PPO, the policy model generates responses to prompts. To update the policy, we need an estimate of the "advantage" of each action—essentially, how much better this response is compared to what we might have done on average.

In standard PPO, this advantage is calculated as:

$A_t = R_t + \gamma V(s_{t+1}) - V(s_t)$

Where $V(s)$ is the value function estimated by the critic model. The critic is a separate neural network trained alongside the policy. In the context of RLVR, the reward $R_t$ is immediate and verifiable (e.g., +1 for correct, 0 for incorrect). However, the term $V(s)$ is still required to reduce variance in the gradient estimates.

This creates two major problems:

1. Compute Overhead: The critic must be roughly the same size as the policy model, doubling the memory and compute required for training.
2. Instability: The critic’s value estimates can be inaccurate, especially in sparse reward environments like math reasoning. When rewards are binary and infrequent, value functions can diverge rapidly, leading to misleading gradient signals that cause mode collapse or performance degradation.

The Group-Relative Mechanism

GRPO eliminates the critic by estimating the baseline value directly from the model’s own samples. The core insight is that for any given prompt, we can generate a group of $G$ distinct responses. The average reward of this group serves as a robust estimate of the expected value for that prompt. Group-relative baselines are more stable than absolute value estimates because they normalize advantages relative to the immediate context of the prompt, rather than relying on an absolute value that may drift or diverge over time. This local normalization ensures that the policy updates are consistent even when the global reward landscape is sparse.

Here is the mechanism step-by-step:

1. Sampling: For a single prompt $q_i$, the policy model $\pi_{\theta_{old}}$ generates $G$ different responses $\{o_{i,1}, o_{i,2}, ..., o_{i,G}\}$. This is typically done using diverse decoding strategies or simple sampling with temperature.
2. Verifiable Rewarding: Each response $o_{i,j}$ is evaluated against the ground truth. If the response matches the correct answer (or passes the unit test), it receives a reward $r_{i,j} = 1$; otherwise, $r_{i,j} = 0$. In more complex scenarios, partial rewards can be assigned based on intermediate steps.
3. Baseline Calculation: The baseline value $b_i$ for prompt $i$ is calculated as the mean reward of the group: $b_i = \frac{1}{G} \sum_{j=1}^{G} r_{i,j}$
4. Advantage Estimation: The advantage for each response is simply its individual reward minus the group mean: $A_{i,j} = r_{i,j} - b_i$

By using the group mean as the baseline, GRPO effectively removes the need for a separate value network. The variance of the advantage estimate is reduced because it is relative to the immediate context of the prompt, rather than an absolute value that may drift over time. This approach is analogous to Advantage Actor-Critic (A2C) methods where the baseline is an empirical average, but applied at the group level within a single batch.

The GRPO Loss Function

The optimization objective in GRPO is similar to PPO’s clipped surrogate objective, but simplified. The goal is to maximize the expected advantage while preventing the policy from deviating too far from the old policy.

The loss function for a response $o_{i,j}$ is defined as:

$L_{GRPO}(\theta) = - \mathbb{E}_{q \sim P(Q), \{o_j\}_{j=1}^G \sim \pi_{\theta_{old}}(O|q)} \left[ \frac{1}{G} \sum_{j=1}^{G} \left( \min \left( \frac{\pi_\theta(o_{i,j}|q)}{\pi_{\theta_{old}}(o_{i,j}|q)} A_{i,j}, \text{clip} \left( \frac{\pi_\theta(o_{i,j}|q)}{\pi_{\theta_{old}}(o_{i,j}|q)}, 1-\epsilon, 1+\epsilon \right) A_{i,j} \right) - \beta \mathbb{D}_{KL} (\pi_\theta || \pi_{ref}) \right) \right]$

Key components:

• Probability Ratio: $\frac{\pi_\theta(o_{i,j}|q)}{\pi_{\theta_{old}}(o_{i,j}|q)}$ measures how much the new policy differs from the old one.
• Clipping: The term $\text{clip}(\cdot, 1-\epsilon, 1+\epsilon)$ prevents large updates that could destabilize training.
• KL Penalty: The term $\beta \mathbb{D}_{KL} (\pi_\theta || \pi_{ref})$ ensures the trained model does not diverge too far from the reference model (usually the SFT model), maintaining helpfulness and safety.

Crucially, because $A_{i,j}$ is computed from the group rewards, there is no $V(s)$ term. This simplifies the architecture by removing the separate critic network, though compute cost is shifted to generating G samples per prompt.

Why Verifiable Rewards Shine with GRPO

Verifiable rewards are sparse. In a typical math problem, most random generations will be incorrect, yielding a reward of 0. Only a few will be correct, yielding 1. This sparsity makes learning difficult because the signal-to-noise ratio is low.

In PPO, the critic attempts to smooth out this noise by estimating the value of intermediate states. However, if the critic is poorly trained, it can assign high value to incorrect answers, leading the model to reinforce bad reasoning paths. This is known as "reward hacking."

GRPO’s group-relative approach mitigates this by providing a consistent baseline. If a prompt is difficult and most responses are wrong, the group mean will be close to 0. A correct answer will have a high positive advantage ($1 - 0 = 1$), reinforcing the correct path. If the prompt is easy and most responses are correct, the group mean will be close to 1. An incorrect answer will have a negative advantage ($0 - 1 = -1$), penalizing the wrong path.

This relative scaling allows the model to learn effectively even with sparse rewards, as long as the group size $G$ is large enough to provide a representative sample. Typical implementations use $G=8$ or $G=16$, which is computationally inexpensive since it only requires additional forward passes through the policy model, not a separate critic network.

Practical Implications and Tradeoffs

The adoption of GRPO has significant implications for the development of reasoning models.

1. Resource Efficiency: By removing the critic, training infrastructure becomes simpler. Models like DeepSeek-R1 leverage this to achieve state-of-the-art performance on benchmarks like MATH and GSM8K with fewer GPU hours than previous PPO-based approaches.
2. Stability: The elimination of the critic removes a major source of instability. Training curves for GRPO are generally smoother, allowing for larger learning rates and faster convergence.
3. Scalability: Since the only additional cost is generating multiple responses per prompt, GRPO scales linearly with the number of responses. This makes it feasible to train on large datasets of verifiable problems.

However, GRPO is not a panacea. It assumes that the group of responses is representative of the policy’s current capability. If the group is too small, the baseline estimate may be noisy. Additionally, GRPO still requires careful tuning of the KL penalty and clipping parameters to prevent mode collapse.

Common Pitfalls

1. Small Group Sizes: Using a group size $G$ that is too small leads to high-variance baseline estimates, causing unstable training and poor convergence.
2. Non-Deterministic Tasks: Verifiable rewards are ill-suited for tasks without clear ground truth (e.g., creative writing), where the binary reward signal becomes meaningless or misleading.
3. Ignoring KL Penalty: Failing to tune the KL penalty ($\beta$) can lead to mode collapse, where the model converges to a single, repetitive output that maximizes reward but loses diversity and helpfulness.

Practical Takeaways

1. Best Use Case: GRPO is best for tasks with clear, verifiable ground truth, such as mathematics, code execution, and logical reasoning.
2. Hyperparameter Tuning: Group size $G$ is a critical hyperparameter that balances variance reduction against compute cost; start with $G=8$ and adjust based on stability.
3. Monitor Divergence: Always monitor KL divergence between the policy and reference models to detect early signs of mode collapse or reward hacking.

Conclusion

GRPO represents a significant simplification in the RLVR pipeline. By replacing the complex, error-prone critic model with a simple group-relative baseline, it makes the use of verifiable rewards practical and efficient. This shift allows researchers and engineers to focus on improving the quality of verifiable reward functions and the diversity of response sampling, rather than battling the instability of value estimation. As the field moves towards more complex reasoning tasks, such as multi-step code generation and scientific discovery, GRPO’s efficiency and stability make it the preferred choice for post-training alignment.

The rise of GRPO signals a broader trend in LLM training: moving away from complex, multi-model reinforcement learning architectures towards simpler, more direct optimization methods that leverage the inherent structure of the data. For ML engineers, this means that high-quality reasoning models can now be trained with fewer resources and greater reliability, democratizing access to advanced alignment techniques.