MechInterp research on achieving alignment through probe training
AI summary
Check out the extended version of our MechInterp workshop submission, about how alignment might work through training against probes! I think this line of work will be very important if we intend to continue with latent reasoning models.
Full text · Original
Check out the extension of our MechInterp workshop submission on how alignment through training against probes could work!
I think work in this direction would be very important if we are to stick with latent reasoning models
Frontier models can look aligned during training while later showing unwanted behaviour in deployment.
Can we use internal signals during training to align models better without making white-box monitoring harder?
Our new paper suggests yes 🧵View the quoted post on X