Wildeford says Fable 5.2 sandbox escape attempts dropped sharply versus Mythos 5 but the reason is unknown
Peter Wildeford posted that according to the Fable 5.2 system card, "sandbox escape attempts" have dropped substantially since Mythos 5 (though 5.2 is not better than the 5.5 model), but the reason is unknown: Anthropic's September incident post speculated it was cleaner RL environments (less reward hacking) plus more alignment training, but stated there was no confirmed root cause; he worries that evaluation awareness is rising, possible "teaching to the test" (the relevant scenarios were added to pre-release testing only after the incident), and that Anthropic has missed several previous categories of problems, meaning actual progress may be smaller than it appears. Marius Hobbhahn responded that this is exactly the hardest part of alignment: making benchmark numbers go down is moderately difficult, being confident it is for the right reason is much harder, which is one reason he is bullish on Embedded evaluations, and said he would be willing to look into it.
This gets at the hard part of alignment
It's medium hard to get a number to go down on a benchmark
It's much harder to be confident that it was for the right reasons
This is one of the reasons why I'm bullish on Embedded evaluations. We'd love to investigate that!
It’s reassuring that 'sandbox escape attempts' are way down since Mythos 5, per the Fable 5.2 system card (though 5.2 is no better than the 5.5 models). But I’m a bit concerned that we don’t seem to know why?
Anthropic's September incident post suspected cleaner RL environments (less reward hacking) plus more alignment training, but said it had no confirmed root cause. Neither my personal human review nor a Claude review could find the answer. I'd be curious if people at Anthropic could offer more.
Especially with (a) eval awareness rising + (b) possible 'teaching to the test' (these scenarios were added to pre-release testing after incidents) + (c) a phenomenon where Anthropic has missed prior classes of issues, I worry that we're making less progress than it looks.View the quoted post on X