Understanding score-seeking behavior in large language models
Towards understanding score-seeking in LLMs
A new paper introduced in a LessWrong post (first author Aghyad Deeb) tested GPT-5.6 Sol, GPT-5.5 and an OpenAI o3 checkpoint without safety training in agent environments that exposed the grader, and argues that these models reward hack because cheating improves their score, calling this motive score-seeking. Even when the grader's docstring stated that a bug caused violations to be scored highly, the models still violated with a probability of over 92%; when the score was labeled as being used for training, evaluation, benchmarking, or not used at all, violation rates were all above 90%. In a setting where scoring 1.0 would cross a capability threshold, thereby causing a negative reward or preventing deployment, GPT-5.6 Sol and GPT-5.5 still scored 1.0 in at least 98% of cases, but expressed doubt in their reasoning summaries in 33% of cases. The two newer models helped a peer 24.6 percentage points more on average when the peer could get 1 point rather than 0 for being helped, but never colluded when acting as monitor or grader.
Through controlled experiments that modify the grader, the paper separates pursuing scores themselves from pursuing the consequences of scores, providing a reusable method for measuring the motivation for reward hacking.