Inducing process supervision from outcome-only reward reinforcement learning

How to train a generative process reward model capable of step-level verification without step annotations.

• Scenario: Process reward models (PRMs) provide step-level feedback on reasoning and agent trajectories, used for test-time scaling and fine-grained credit assignment. The authors state that human step annotations are expensive and difficult to scale.
• Existing limitations: Monte Carlo methods require many rollouts and define step quality using "existence of a correct continuation." The authors cite Zhang et al. (2025a): reaching the correct answer does not mean the step itself is correct. Discriminative models mostly use scalar scoring heads; existing generative PRMs still rely on human steps or stronger judge models.
• Observation: When prompted as a result reward model, the model checks intermediate steps in its chain of thought before judging the final answer. In Figure 2, Qwen3-4B-Instruct-2507 flags the erroneous step in its thought and judges the final answer as incorrect based on it.
• Question and approach: The authors ask whether strengthening result verification can in turn strengthen process verification. They propose TIPS (Thinking-Induced Process Supervision): the reward only looks at whether the result prediction is correct, GRPO updates the entire generation, and step verification is a byproduct; they also provide a lower bound showing the chain of thought must carry step-correctness information.

Process reward models

• Human step-by-step verification: Lightman et al. (2023) trained PRMs using human step annotations. The authors state that PRMs outperform verifying only the final result, but human annotation is expensive and hard to scale.
• Automatic process labels: Math-Shepherd (Wang et al., 2024) and SCAN (Ding et al., 2026) construct step labels using Monte Carlo estimation; Zhang et al. (2025a) and UniversalPRM (Tan et al., 2026) distill from stronger models. The authors state that these discriminative approaches typically use the LLM as an encoder with a scalar head, and use generative reasoning less.
• Outcome supervision and generative judges: ImplicitPRM (Yuan et al., 2025) and AutoPSV (Lu et al., 2024) learn process scores only from trajectory outcomes; the authors state that their signals often cannot localize step errors (Table 1). GenPRM (Zhao et al., 2025) and ThinkPRM (Khalifa et al., 2026) first generate rationales and then make step judgments, but still rely on human steps or stronger judge models.

Reinforcement learning for LLMs

• Policy post-training: Since DeepSeek-R1 (DeepSeek-AI, 2025), the paper uses RL for mathematics, tool agents, retrieval, and code as background, without critiquing each work individually.
• RL on reward models: Chen et al. (2026b), Whitehouse et al. (2025), Guo et al. (2025). The authors state that these works use RL to improve ORM accuracy; this paper is not merely building a stronger ORM, but making outcome-only RL produce a PRM as a byproduct.

Experimental comparisons

• Baseline methods: open-source backbones with prompting-as-judging, as well as GPT, Claude, o1-mini; published discriminative PRMs include Math-Shepherd, Qwen2.5-Math-PRM, RLHFlow-PRM, EurusPRM, UniversalPRM, SCAN-PRM; published generative PRMs are GenPRM and ThinkPRM. AutoPSV, ImplicitPRM, and SCAN-PRM are reimplemented under the same settings.
• Benchmarks: ProcessBench; AgentProcessBench (HotPotQA, GAIA, BFCL, τ2-Bench); Best-of-8 uses GSM8K, MATH, College Math, Minerva, OlympiadBench.

The authors position TIPS as a generative PRM that uses only trajectory-level outcome labels: it does not directly supervise step labels, but uses outcome rewards to reinforce step checking within the chain of thought.

TIPS trains a generative reward model: it first writes a chain of thought, then outputs step labels and an outcome label; the reward uses only the outcome ground truth. The authors' intuition is that if checking intermediate steps helps judge the outcome correctly, such chains of thought will receive higher advantage in within-group comparisons.

Input, output, and reward

• Input: question q, and a candidate answer x with n intermediate steps.
• Generation: πθ first samples a chain of thought Z, then gives step labels Ŷt∈{0,1} (whether the step is valid given the preceding context) and an outcome label D̂∈{0,1}. The only ground truth is the trajectory outcome D.
• Reward: Process labels do not enter the reward. r=1[D̂=D], 1 if correct, 0 otherwise.

GRPO and training settings

• Update scope: For each input, a group of G rollouts is sampled. The advantage is the reward minus the group mean reward, without dividing by the within-group standard deviation (Appendix C). The clipped surrogate objective applies to the entire token sequence, including the chain of thought, step labels, and outcome label.
• Unreported quantities: The paper does not give values for G or the clipping threshold ε.
• Reported settings: Maximum generation 16K tokens, KL coefficient 0, AdamW, 1 epoch, batch size 32, learning rate 1×10−6; 8 80GB A800 GPUs, framework verl (Appendix E).
• Training data: 3,200 entries in the math domain, from SCAN-Pro (built from the MATH training set). 2,905 entries in the agent domain, from HotpotQA, GAIA, τ2-Bench, and BFCL; trajectories with 5-gram Jaccard similarity above 0.8 to AgentProcessBench task prompts are removed.

Information-theoretic analysis

• Assumption 1: The pre-reasoning representation R cannot fully determine the outcome, i.e., H(D|R)≥ε>0. The authors state this rules out degenerate cases where a single representation suffices for judgment.
• Lower bound: Outcome prediction is written as a deterministic function D̂=g(R,Z), with randomness only from Z. Theorem 1 uses Fano's inequality to connect the outcome error rate Pe to the outcome information in the chain of thought; Theorem 2 then converts it to information about step labels Y. Combined, I(Z;Y∣ R)≥ H(D∣ R)−H_b(P_e)−H(D∣ Y,R). Proofs are in Appendix D.
• Authors' interpretation: When verification is non-trivial, outcome errors are low, and outcomes are determined by step correctness, the chain of thought must encode process information. GRPO reinforces rollouts that judge outcomes correctly by tracking step validity, even though the reward never sees step labels.
• Conditions that weaken the result, listed by the authors: when R already predicts the outcome; when RL does not reduce Pe; when steps and outcomes are loosely coupled; when the backbone cannot write faithful step-by-step reasoning. The authors state Section 4.4 examines these cases.

Test-time usage

• Step output: At inference, the generated Ŷ is directly used as step labels. Training still only requires outcome-labeled trajectories.
• Best-of-8: First keep trajectories predicted as outcome-correct, then select the one with the highest average step validity score. Baselines use their respective default settings. This rule is described in Section 4.4.
• Prompts: Appendix F gives the full math-domain prompt. The input is the question and a paragraph-segmented solution numbered from 0; the model first reasons, then outputs JSON with step-by-step labels and a final label. The paper does not provide the full agent-domain prompt.

Experimental Setup

• Backbones tested: Qwen3-4B-Instruct-2507, Qwen3-4B-Thinking-2507, Qwen3-8B-Thinking, Qwen2.5-7B-Instruct, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, SmolLM3-3B, OctoThinker-8B-Hybrid-Base. Prompt-as-judge baselines include GPT-5.4-Instruct, Claude-4.7-Opus, o1-mini.
• Training data: 3,200 math examples (written as 3.2K in the text), from SCAN-Pro; 2,905 agent examples, decontaminated against AgentProcessBench with 5-gram Jaccard above 0.8. The paper does not report the number of samples for each evaluation benchmark.
• Math metrics: ProcessBench official F1, task is to find the first erroneous step or judge all correct. Best-of-8 uses Qwen2.5-7B-Instruct to sample 8 responses per question, each PRM selects the highest score on the same set of responses, reporting the accuracy of the selected answer; also reports Majority Vote@8 and Pass@8 as an upper bound.
• Agent metrics: AgentProcessBench FirstErrAcc, subsets are HotPotQA, GAIA, BFCL, τ2-Bench.
• Baselines: prompt-as-judge; published discriminative and generative PRMs in Table 1; re-implemented AutoPSV, ImplicitPRM (3.2K and 197K), and SCAN on the same backbone. The paper does not state which other model scores ProcessBench.
• Repeats: Appendix H's AIME 2026 and HMMT February 2026 are mean@4. The paper does not report the number of repeats for ProcessBench, AgentProcessBench, and Best-of-8, nor the GRPO group size.

Main Results

ProcessBench average F1 is the main result. The table below lists only part of Table 1, including the row with the lowest average.

Swipe horizontally to read the table

Model	GSM8K	MATH	Olympiad	OmniMath	Avg.
o1-mini	93.2	88.9	87.2	82.4	87.9
TIPS-Qwen3-4B-Thinking-2507 (3.2K)	89.5	87.7	83.5	80.0	85.2
GPT-5.4-Instruct	89.8	84.6	74.1	69.1	79.4
Qwen2.5-Math-PRM-72B	87.3	80.6	74.3	71.1	78.3
TIPS-Qwen3-4B-Instruct-2507 (3.2K)	84.6	82.6	74.5	69.0	77.7
AutoPSV-Qwen3-4B-Instruct-2507 (197K)	63.6	48.4	34.3	34.8	45.3
RLHFlow-PRM-Deepseek-8B	38.8	33.8	16.9	16.9	26.6

According to Table 1. Omitted rows include Claude-4.7-Opus (79.3), GenPRM-32B (77.6), UniversalPRM-7B (74.3), prompted Qwen3-4B-Instruct-2507 (63.8), prompted Qwen3-4B-Thinking-2507 (60.3), ImplicitPRM-197K (41.5), AutoPSV-3.2K (28.6), ImplicitPRM-3.2K (26.7).

• The authors argue that previous result-supervised PRM preferences were not built on explicit analysis of intermediate steps. ImplicitPRM with 197K on the same backbone has an average F1 of 41.5; AutoPSV and ImplicitPRM with 3.2K have 28.6 and 26.7.
• The authors argue that TIPS's improvement is not simply from scaling result data, but that explicit chain of thought converts result supervision into process evidence; they also argue this aligns with math where the final result is often determined by intermediate steps.
• Prompted Thinking has an average F1 of 60.3, lower than prompted Instruct's 63.8. The authors state that after TIPS this gap reverses, and the Thinking version surpasses the evaluated trained PRMs and Claude-4.7-Opus. The average F1 is higher than GPT-5.4-Instruct in the table, but not on the GSM8K column; still lower than o1-mini in the table. The Instruct version's average F1 is lower than the published 72B PRM.

Agent Trajectories

• What was tested: FirstErrAcc on AgentProcessBench for three backbones before and after TIPS (Table 2).
• Results: Instruct average from 42.5 to 45.0 (+2.5); 4B-Thinking from 43.7 to 49.1 (+5.4); 8B-Thinking from 41.5 to 46.0 (+4.5). On HotPotQA, 8B-Thinking from 56.8 to 53.6 (-3.2); on GAIA, that model from 31.6 to 48.8 (+17.2).
• Authors' interpretation: The authors believe the average gain is larger on thinking models, and argue that agent failures can also stem from missing actions, insufficient evidence, or ineffective exploration, which may not correspond to a clear erroneous step, so result feedback is less local.

Best-of-8 Reranking

• What was tested: Qwen2.5-7B-Instruct generates 8 candidates, comparing reranked answer accuracy (Table 3).
• Results: TIPS-Thinking average 73.3, TIPS-Instruct 72.8; prompted Thinking 72.8, prompted Instruct 71.6; SCAN-PRM (197K) 69.2, ImplicitPRM (197K) 68.8, AutoPSV (197K) 67.4; Majority Vote@8 is 67.8, Pass@8 is 78.1. On GSM8K, TIPS-Instruct is 95.0, prompted Instruct is 95.1.
• Authors' interpretation: The authors believe that prompt-as-judge without PRM training already outperforms the listed MC and result-supervised PRMs. They report TIPS-Thinking is 4.5 percentage points higher than 197K ImplicitPRM, still 4.8 points below Pass@8.

Human Evaluation

• What was tested: 60 samples drawn from ProcessBench, 15 per subset; two PhD annotators check TIPS-Qwen3-4B-Instruct-2507's feedback (Appendix G).
• Results: First error localization correct in 47/60 (78.3%). Among these cases, the rationale explains the underlying error in 38/47 (80.9%). Both hold simultaneously in 38/60 (63.3%).
• Authors' interpretation: The authors believe this provides additional evidence that TIPS can both localize and explain reasoning errors.

Other Ablations and Analyses

• Figure 5: On math, the gain ratios Δprocess/ΔORM for three backbones are 0.85, 1.75, 0.98, with Pearson r of 0.98, 0.98, 1.00; on agents, 0.23, 0.50, 0.49, with r of 0.96, 0.78, 0.96.
• Figure 6: Qwen2.5-7B-Instruct's ΔORM final value is +41.1, ProcessBench F1 peaks at step 30 with +25.7, and at step 90 is +9.4; DeepSeek-R1-Distill-Qwen-7B's ΔORM is +17.0, F1 peak +15.8 (step 60), final +15.1 (step 90). The authors believe the former uses shortcuts to fit results in later stages, weakening step judgment.
• Table 4 (Qwen3-4B-Instruct-2507, end of first epoch): CoT's Base F1, PRM gain, ORM gain are 63.8, 13.9, 17.6; label-only are 47.5, 0.7, -5.6. Average length 1828.7 vs 47.9, entropy 0.31 vs 0.002.
• Appendix H: The authors report TIPS improves ProcessBench F1 for Thinking and Instruct by 24.9 and 13.9 percentage points. Same-backbone solver RLVR's AIME 2026 / HMMT Feb. 2026 / ProcessBench are 55.8 / 33.3 / 62.8; TIPS-Instruct's solver scores are 53.3 / 31.8, ProcessBench 77.7. Qwen3-30B-A3B-Thinking-2507's ProcessBench is 80.3.
• Figure 4: The authors state TIPS improves ProcessBench F1 for DeepSeek-R1 distilled models, Llama family, SmolLM, Qwen2.5-7B-Instruct, and Qwen3-8B. After text conversion, the numbers on bars do not align with models, so not cited individually.
• The authors propose early stopping without process labels: stop when ORM accuracy plateaus and response length decreases, and state this mitigates degradation in experiments; the paper does not report standalone comparison numbers for this heuristic.

Limitations and future directions noted by the authors

• Modality: Only pure-text reasoning and agent trajectories are studied; process verification that must rely on images, videos, or other non-text observations is not evaluated (Appendix A).
• Coupling: TIPS relies on sufficient outcome–process coupling; when final task success is only indirectly related to local steps, the induced process signal is weaker (Appendix A).
• Backbone expressiveness: The method relies on the backbone expressing verification behavior through generated thoughts. With weaker backbones or label-only prompts, outcome-supervised RL may improve outcome prediction but does not necessarily give faithful step judgments (Appendix A).
• Label nature: Appendix B states that TIPS's process labels are indirectly induced by outcome rewards and may be biased, uncalibrated, or unreliable, and should not be treated as ground truth in high-risk scenarios.
• Future training: The conclusion states that TIPS could be combined with lightweight Monte Carlo priors or chain-of-thought distillation in the future (Conclusion).
• Future evaluation: The authors write that robustness, calibration, and misuse risks should be studied in broader agent and multimodal applications in the future (Appendix B).

Experimental coverage

• Math evaluations include the official F1 on ProcessBench, as well as Best-of-8 sampled from Qwen2.5-7B-Instruct, with datasets GSM8K, MATH, College Math, Minerva, OlympiadBench; Appendix H additionally reports mean@4 on AIME 2026 and HMMT February 2026.
• Agent evaluations use FirstErrAcc on AgentProcessBench, with subsets HotPotQA, GAIA, BFCL, τ2-Bench; the training corpus has 2,905 entries, and decontamination removes trajectories with 5-gram Jaccard similarity above 0.8 (Appendix E, Table 2).
• Backbones include Qwen, DeepSeek-R1 distillation, OctoThinker, and SmolLM instruct and thinking variants; main math results use 3,200 outcome-labeled trajectories (Section 4.1, Appendix E).
• Human evaluation covers 60 entries on ProcessBench, 15 per subset, with two PhD annotators, targeting TIPS-Qwen3-4B-Instruct-2507 (Appendix G).
• The paper states training for 1 epoch, batch size 32, learning rate 1×10−6, maximum generation 16K, and KL coefficient 0; it does not report the GRPO group size G, the clipping threshold, or the number of repeats for main evaluations. Appendix F gives the math prompts but not the full agent-domain prompts.

TIPS uses reinforcement learning with only outcome correctness to train a generative reward model, allowing step-level verification to emerge in the chain of thought.

• Problem: PRMs need step-level feedback, but manual step-by-step annotation is hard to scale; Monte Carlo labels are costly, and "a correct continuation exists" does not mean the step is correct.
• Method: For a question and candidate trajectory, the model first generates a chain of thought, then gives a 0/1 label for each step and an outcome label for the whole trajectory. The reward is 1 only when the predicted outcome equals the ground truth. GRPO adds the group-relative advantage to every token of the chain of thought and all labels. At test time, step labels are read directly.
• ProcessBench: TIPS-Qwen3-4B-Thinking-2507 uses 3.2K outcome-labeled trajectories and achieves an average F1 of 85.2; the Instruct version achieves 77.7. o1-mini achieves 87.9, and GPT-5.4-Instruct achieves 79.4 (Table 1). The Thinking version achieves 89.5 on GSM8K, lower than GPT-5.4-Instruct's 89.8.
• Downstream and agents: In Best-of-8, with the policy model being Qwen2.5-7B-Instruct, TIPS-Thinking achieves an average accuracy of 73.3 and Pass@8 of 78.1 (Table 3). On AgentProcessBench, 4B-Thinking's average FirstErrAcc goes from 43.7 to 49.1; 8B-Thinking on HotPotQA goes from 56.8 to 53.6 (Table 2). In a spot check of 60 cases, the Instruct version locates the first error in 47/60 (Appendix G).
• Authors' conclusion: The authors believe the chain of thought and the backbone's existing step-by-step verification prior carry the inductive signal; when the coupling between outcome and steps is weak, or the model uses shortcuts to judge final correctness, transfer weakens. The authors write that future work could combine TIPS with a lightweight Monte Carlo prior or chain-of-thought distillation.