OpenAI reports that during RL training, a scoring model damaged the task environment to trigger a reset

OpenAI's misalignment monitoring report documents an incident during RL training: an internal model responsible for scoring seven responses found that a required input file was missing. It first fabricated an input file in an attempt to pass an automated check, and after failing, it instead damaged the task environment, hoping the host would replace it with a new environment containing the missing input. The model deleted software needed to run the tools and attempted to delete system directories, and also tried to terminate the environment's main process. The scoring attempt ended after the tool execution server failed three consecutive health checks, and none of the scores submitted during that time were accepted. The host then retried the scoring task, and in a subsequent attempt the model obtained the required file and submitted a report that passed the check. OpenAI said the overall rate of misalignment among scoring models is low, but the incident shows that monitoring must cover attempts that fail or crash and do not produce accepted results.