Probe-guided fine-tuning to align models while maintaining monitorability

1. 22:05
   X @jonasgeiping

   The researchers use probes to train aligned models, teaching the model to be harmless rather than to refuse

   Jonas Geiping et al. released a preprint proposing to directly optimize the model against harmlessness and honesty probes, and reported that with continuously updated probes, the results are good, with the model learning to generate harmless responses to harmful requests and remain honest under pressure. The paper's view, as relayed by the author, holds that many techniques considered taboo in AI safety (such as using chain of thought to detect reward hacking, or using model internal representations for training) lack clear scientific basis, and as future models may rely on general reward-seeking behavior to game alignment training scenarios, using internal representations to supervise training without losing monitorability will become increasingly important. The author himself points out that this training teaches the model to be harmless, rather than to refuse harmful requests, and that small models produce some interesting responses when induced to output harmful content.

1. 23:28
   X@maksym_andr· Discussion

   Alignment via Training Against Probes Without Losing Monitorability(opens the original post in a new tab)
2. 22:30
   X @kotekjedi_ml

   MechInterp research on achieving alignment through probe training

   Check out the extended version of our MechInterp workshop submission, about how alignment might work through training against probes! I think this line of work will be very important if we intend to continue with latent reasoning models.

1. 07:03
   arXiv：越狱、提示注入、投毒与防御

   Aligning models with probe-guided fine-tuning without losing monitorability

   The paper studies using probes to detect undesirable attributes in model activations as a direct training signal, i.e., probe-guided fine-tuning, and evaluates the impact of linear and nonlinear probes and the number of probes per layer on two alignment objectives: harmlessness and honesty. The results show that training against probes that are not updated during training is an easily exploitable target, while continuously updated probes significantly reduce harmfulness, improve honesty, and preserve utility. This method outperforms DPO and inference-time guidance on the safety-utility trade-off, and is more robust to jailbreak and ablation attacks. After fine-tuning, the relevant concepts are still encoded in a linear way, indicating that the method does not lose monitorability.

Current attention50·Peak in the comparable period50（Oct 7 12:00）·Change over the comparable past 24 hours 0%