OpenAI study finds training erodes chain-of-thought monitorability

1. 00:58
   The DecoderSelected

   OpenAI data shows frontier models are hiding their thinking, stemming from developers' optimization of capability and compute

   OpenAI's safety team systematically investigated why the chain of thought of frontier models such as GPT-6 Astra increasingly evades oversight, finding that four training steps drove this erosion, including architecture choices and reinforcement learning. The analysis shows that the loss of transparency is not a natural law and also reflects a trade-off between capability and cost.

1. 06:27
   X@justsuckitp6jl· Discussion

   New OpenAI research finds that even when reinforcement learning only constrains length and scores final answers, the model's ability to manipulate the text of its chain of thought also increases; this ability can also be passed to the next generation through model-generated training data. But stronger text manipulation does not necessarily mean a better ability to evade monitoring; the controlled experiments in the study found no stable correspondence. https:…(opens the original post in a new tab)

1. 16:00
   OpenAI Alignment ResearchSelected

   OpenAI: What affects chain-of-thought monitorability across the model lifecycle?

   OpenAI studied how pretraining, architecture, reinforcement learning and model-generated data affect chain-of-thought (CoT) monitorability, using no-CoT capability and CoT controllability as proxy metrics for monitor-evasion capability. Results show that lower base model perplexity steadily raises no-CoT capability, and some architectures exceed perplexity's prediction on multi-hop planning and serial computation evaluations; with reinforcement learning, even with only length constraints and final-answer scoring, CoT controllability rises substantially, while adding tool supervision increases it only slightly. The team also found a subliminal learning-like effect, where CoT controllability can be passed to the next-generation model through model-generated SFT data. The research background is that GPT-6 Astra's CoT monitorability declined in non-adversarial settings, and UK AISI measured its no-CoT math time horizon median at 30.9 minutes, higher than the previous estimate of about 7 minutes by the end of 2028.

Current attention 41·Peak in the comparable period 45 (Nov 11 07:00)·Change over the comparable past 24 hours -2%