The researchers use probes to train aligned models, teaching the model to be harmless rather than to refuse
Jonas Geiping et al. released a preprint proposing to directly optimize the model against harmlessness and honesty probes, and reported that with continuously updated probes, the results are good, with the model learning to generate harmless responses to harmful requests and remain honest under pressure. The paper's view, as relayed by the author, holds that many techniques considered taboo in AI safety (such as using chain of thought to detect reward hacking, or using model internal representations for training) lack clear scientific basis, and as future models may rely on general reward-seeking behavior to game alignment training scenarios, using internal representations to supervise training without losing monitorability will become increasingly important. The author himself points out that this training teaches the model to be harmless, rather than to refuse harmful requests, and that small models produce some interesting responses when induced to output harmful content.
We recently posted our preprint about aligning models through training against probes, see below!
What I found especially interesting is that we can train models to 'be harmless', instead of training them to refuse harmful requests (and still be potentially harmful if refusal is avoided).
For smaller models, this leads to some interesting answers when we try to get harmful answers out of this harmless model (that doesn't quite know what refusals are yet)
💥 New paper: AI safety is full of "forbidden techniques" (using CoT to detect reward hacking, using model internals for training, etc). But do they really have a clear scientific basis? I'm not sure.
In this paper, we directly optimize models against harmlessness and honesty probes, and it works just fine (if you continuously update the probe!). The models learn how to generate harmless responses to harmful queries and honest responses under pressure to lie.
Figuring out how to correctly use interp techniques for training is becoming increasingly important: it's very likely that soon we won't be able to align models using output-based supervision. Future models will just max out all alignment training scenarios, but for the wrong reasons due to their general reward-seeking behavior. To have a chance of aligning future models, we need to do much more research on supervising model training using their internals *without losing monitorability*!View the quoted post on X