Skip to content
WeChat · Weijiqi Lixin·Indexed by this site · Originally published· AI updates →· Its place in the scan record →SelectedAttention42

New York University study: multi-harness reinforcement learning gains may just be adaptation to the training framework

纽约大学:刷分假象被戳穿,多 Harness 强化学习,可能只是适配框架,没有学到真本事

AI summary

A controlled-experiment paper from New York University finds that much of the gain from multi-harness reinforcement learning is merely configuration adaptation to the execution framework used in training and does not translate into transferable coding capability. The experiment uses the SFT version of Qwen3-8B as a unified starting point, keeps trajectory data, update steps, rewards and loss function consistent, and changes only the GRPO grouping rule, comparing each task forming its own group within each harness against putting trajectories from Aider, OpenHands, SWE-agent and Qwen Code into the same group for cross-framework reward comparison. On the minimal weak-ReAct harness that did not appear in training, cross-framework pooling is only 0.25 percentage points higher than within-group comparison, with a 95% confidence interval of [-0.48, +1.02], and the winner between the two groups flips after changing the random seed. The study also shows that for the same model, switching to a different harness can change the SWE-bench solve rate from 2.14% to 9.27%, a 4.3-fold difference, larger than the 1.16-fold fluctuation brought by the RL training recipe.

Why it was selected

The paper uses controlled experiments to separate exposure and reward grouping in multi-harness training, and provides comparison data showing that switching frameworks can make SWE-bench solve rates differ by 4.3 times.

Read originalmp.weixin.qq.com