Generalization of gradient routing to unrouted reward hacking types
Gradient Routing Generalizes to Unrouted Hack Types
The authors conduct extended experiments on the gradient routing method of Ward, Hu et al (2026), routing gradients only for one type of reward hacking (the visible set), to test whether it can suppress other types of hacking that the model also learned in the SFT data (the invisible set), simulating the situation where a real monitor can only catch some hacking types during training. The experiments are based on Qwen3-32B and the TerminalWrench agent coding trajectory dataset, with 12 visible/invisible splits in total, compared against FLOOR, which randomly routes 20% of hacking, and CEILING, which does no routing. Results: in 8 of the 12 splits, transfer was not significantly lower than 1, better than the authors' and Claude's prior predictions; but the Python, Code vs ops, and Serious grader tampering groups showed poor transfer, and the Hacky reasoning group showed 15% silent hacking (FLOOR and CEILING were about 0%), and the authors state they cannot yet explain these failures.