Why did frontier reasoning models improve so fast? Because labs discarded subjective human preference models and learned value functions. In their place, Group Relative Policy Optimization (GRPO) and Verifiable Rewards (RLVR) turn compilers, test runners, and symbolic solvers into automated engines of self-improving intelligence.
Adjust the verifier fidelity, optimization algorithm, rollout group size ($G$), and KL divergence regularization. Observe how accuracy, VRAM footprint, gradient variance, and reward hacking risks shift dynamically.
Compilers and unit tests provide binary ground truth ($r \in \{0, 1\}$). Group size $G=8$ balances sample variance with latency.
Inspect the exact structural inputs, mathematical core operations, and policy gradient outputs of a single GRPO update step.
[q₁, q₂, ..., q_B]G = 8 per promptDocker / Pytest / Lean4π_ref (frozen base)Queries must supply deterministic verification criteria (e.g. input/output test assertions, compiler flags, or theorem statements).
(B × G, Sequence_Len)r_i = Verifier(q, o_i)μ = mean(r), σ = std(r)A_i = (r_i - μ) / σmin(ρ A, clip(ρ) A) - β D_KLNo Value Network $V_\phi$ is ever invoked. The advantage is fully computed from intra-group rollout competition.
1.0× (Critic-Free)∇_θ J_GRPO(θ) via AdamWBinary pass/fail verified<think> ... </think>Emits updated weights $\pi_{\theta}$ biased toward trajectories that autonomously check their own intermediate deductions.
Follow a single training step from sampling candidate reasoning paths to computing clipped policy gradients without a value network.
Compare the three dominant policy optimization paradigms across GPU memory footprint, online exploration, reward stability, and reasoning capacity.
Does RLVR teach models fundamentally new deductive capabilities, or does it merely compress search over pre-existing weights?
Critics argue that RLVR cannot inject new knowledge that wasn't present during pretraining. In this view, RLVR merely increases the likelihood of sampling correct reasoning paths that already had small non-zero probabilities in the base model.
Recent findings (Wen et al., ICLR 2026) using the CoT-Pass@K metric demonstrate that prolonged RLVR does more than compress search: it fundamentally reorganizes token dependency graphs into self-verifying loops.
In supervised learning, the loss function itself must be mathematically differentiable. But an external sandbox running pytest, a compiler, or an SMT solver produces a discrete binary exit code (0 or 1) with zero or undefined derivative. How does PyTorch backpropagate this?
The Intuition: The sandbox score acts as an external volume dial. If tests pass ($A_i > 0$), gradient descent increases the likelihood of emitting those exact tokens; if tests fail ($A_i < 0$), it suppresses their probability. The likelihood ratio serves as the universal adapter bridging arbitrary, black-box execution environments with continuous neural network weights.
Where does RLVR break down? Rigorous engineering demands knowing when programmatic verifiers fail or create unintended side effects.
Mathematically proven elimination of value network; $A_i = (r_i - \mu)/\sigma$ stabilizes policy gradients on math datasets.
DEEPSEEKMATH ARXIV ↗Demonstrated that rule-based verifiers without SFT induce self-correction and internal chain-of-thought exploration.
DEEPSEEK-R1 REPORT ↗Established that prolonged RLVR extends reasoning boundaries beyond static search compression.
ICLR 2026 PAPER ↗Formulated verifiable checklists from knowledge graphs to mitigate reward sparsity in non-coding domains.
K2V RESEARCH ↗“GRPO succeeded because it made reinforcement learning computationally realistic for LLMs by deleting the Critic. RLVR succeeded because it replaced fallible human judgment with programmatic ground truth.”
The transition from RLHF to RLVR represents AI's shift from subjective aesthetic imitation to objective computational correctness. Together with dynamic reasoning budgets, it is the primary engine behind the frontier model leap.