HKU NeurIPS 2026 paper reveals GRPO's implicit advantage symmetry; A-GRAE balances accuracy and exploration
你的模型越训越「不敢想」:港大NeurIPS 2026揭露GRPO隐藏的双重枷锁
The team of Qu Liangqiong at the University of Hong Kong (first author Yu Zhiqi) proposes in a NeurIPS 2026 paper the perspective of "implicit advantage symmetry," which uniformly explains two major flaws of GRPO: positive and negative advantages within a group sum to zero, so unsampled but correct paths receive no direct update, suppressing exploration, and performance under large sampling budgets (e.g., Pass@256) is instead lower than the base model; the sample update magnitude is largest at medium difficulty (p=0.5), and easy and hard problems receive the same weight, ignoring difficulty differences. Based on this, the team proposes Asymmetric Group Relative Advantage Estimation, A-GRAE, which decays and suppresses the advantages of correct trajectories and dynamically implements an easy-to-hard curriculum schedule as training progresses; it is plug-and-play and adds no extra computation. The paper is on arXiv (2602.05548), and the code is open-sourced at https://github.com/HKU-HealthAI/A-GRAE.