Apollo Research member: training against CoT-like surfaces may be similar to training against CoT
Training against CoT-like surfaces may be similar to training against CoT
AI summary
A member of Apollo Research proposed that applying training pressure to CoT-like surfaces (such as code comments) may have an effect similar to training against CoT. They speculated that the model may not have separate features distinguishing "marked as reasoning" from "sounds like reasoning." The authors said that the earlier study Output Supervision Can Obfuscate the Chain of Thought had already made a similar argument and provided empirical support.