What outcome-reward RL buys: finishing on time, reliability, and rare reach
What Outcome RL Buys: Finishing, Reliability, Rarely Reach
Quick read
Based on the paper's full text (AI-generated); refer to the original
Researchers compared 20 released outcome-reward RL models (1.5B to 32B, rewarding only whether the final answer is correct) with their starting thinking models on physics and math under fixed token budgets, decomposing pass@1 gains into three parts: finishing, reliability, and reach. Results show that at 4k and 8k budgets, when the base model often exhausts the budget before answering, 82% of the change comes from finishing; when tokens are sufficient, gains are mainly reliability; while reach, i.e., solving problems the base model cannot solve, contributes at most 2.1 percentage points, and among the 144 physics problems solved only by the RL model that were checked, only 3.5% held up after resampling and verification. A small-scale controlled experiment on Qwen3-VL-8B-Thinking provided the physics setup of the problems in five ways, but did not show transfer to held-out problems beyond plain RL.
- Outcome RL can improve pass@1, but existing work shows that at large k, pass@k instead falls behind the base model, and each pass@1 is scored under some token budget, so a single number mixes three things: finishing on time, answering correctly, and solving new problems. The authors aim to distinguish whether the gains from RL are behavior change or new knowledge, and to test whether supplying knowledge can let the model learn content the base model cannot.
- Precisely decompose the change in pass@1 under a fixed budget into three items: finishing (answering within budget), reliability (answering correctly after answering), and reach (solving problems the base model got wrong all eight times, with the reverse mirror deducted). Read comparisons of 20 released RL models with their thinking base models on physics and math problems, and conduct a small-scale RL probe on an 8B model, supplying the physics setup through five channels.
- When the base model often hits the budget, the change is mainly finishing (about 82% at 4k and 8k); after giving the base model more tokens, it can close most of the gap on its own. When the base model can already finish answering, the gains are mainly reliability, and the steps used were already written in the base model's samples. Reach is at most about 2.1 percentage points across the four sets; in a spot check of 144 physics problems solved only by RL, only 3.5% held up. In a small-scale probe, supplying the setup changed the model during training but did not show transfer to held-out problems.
- The released-model study and small-scale probes have boundaries in scale, sampling, and scoring, and cover only thinking base models; the probe has only one seed, one model family, and one domain per treatment group, and does not test long-duration training or large-scale distillation. The reach definition counts gains from problems the base model never answered correctly but left incomplete samples as behavior; physics problems were filtered by the base model, MATH-500 was not resampled for problems where only one side failed, strict answer matching is stricter for larger models, and part of the reliability gains come from answer format.
The paper provides a method for decomposing pass@1 gains by token budget; the result that reward RL gains come mainly from behavior change rather than new knowledge can serve as a reference for interpreting training reports.