GPT 6.2 Astra system card shows chain-of-thought-free problem-solving capability continuing to rise; researchers say it likely reduces chain-of-thought monitorability
Researcher Tomek Korbak said the most important chart in the GPT 6.2 Astra system card shows chain-of-thought-free problem-solving capability continuing to rise, likely translating into lower chain-of-thought monitorability. System card Figure 56 shows that the accuracy of OpenAI models across generations in solving math competition problems without chain of thought has continued to rise, with 5.6 Sol at over 30%, 6 Astra rising to just over 70%, and the latest GPT 6.2 Astra approaching 80%. The dataset extends Greenblatt with harder high school math competition problems including AIME, HMMT, CMIMC, SMT, BRUMO and OTIS Mock AIME, with error bars as 95% Wilson confidence intervals. According to the UK AISI team post he cited, the team tested GPT 6.2 Astra's ability to evade monitoring and solve problems without reasoning before release, with the report at https://deploymentsafety.openai.com/gpt-6-2-astra-sol/external-evaluation-for-monitorability-uk-aisi
I think this is the most important plot from 6.2 Astra's system card: the trend increasing no-CoT capabilities continues, which is very likely to translate into lower CoT monitorability
My team at @AISecurityInst tested GPT 6.2 Astra before release, assessing its ability to evade monitors and solve problems without showing its reasoning. Read our findings in the system card: https://deploymentsafety.openai.com/gpt-6-2-astra-sol/external-evaluation-for-monitorability-uk-aisiView the quoted post on X