Skip to content
Tomek Korbak· @tomekkorbak · X·Indexed by this site · Originally published· Evaluation →· Its place in the scan record →

GPT 6.2 Astra system card shows chain-of-thought-free problem-solving capability continuing to rise; researchers say it likely reduces chain-of-thought monitorability

AI summary

Researcher Tomek Korbak said the most important chart in the GPT 6.2 Astra system card shows chain-of-thought-free problem-solving capability continuing to rise, likely translating into lower chain-of-thought monitorability. System card Figure 56 shows that the accuracy of OpenAI models across generations in solving math competition problems without chain of thought has continued to rise, with 5.6 Sol at over 30%, 6 Astra rising to just over 70%, and the latest GPT 6.2 Astra approaching 80%. The dataset extends Greenblatt with harder high school math competition problems including AIME, HMMT, CMIMC, SMT, BRUMO and OTIS Mock AIME, with error bars as 95% Wilson confidence intervals. According to the UK AISI team post he cited, the team tested GPT 6.2 Astra's ability to evade monitoring and solve problems without reasoning before release, with the report at https://deploymentsafety.openai.com/gpt-6-2-astra-sol/external-evaluation-for-monitorability-uk-aisi

Full text · Original

I think this is the most important plot from 6.2 Astra's system card: the trend increasing no-CoT capabilities continues, which is very likely to translate into lower CoT monitorability

Quoted postJoseph Bloom@JBloomAus
My team at @AISecurityInst tested GPT 6.2 Astra before release, assessing its ability to evade monitors and solve problems without showing its reasoning. Read our findings in the system card: https://deploymentsafety.openai.com/gpt-6-2-astra-sol/external-evaluation-for-monitorability-uk-aisi
View the quoted post on X

Source: Tomek Korbak · x.com