Skip to content
LessWrong· draganover·Indexed by this site · Originally published· Opinion →· Its place in the scan record →

LessWrong author suggests keeping some alignment techniques for evaluation only

Alignment 'train/test splits' should be sacred

AI summary

LessWrong author draganover argues that alignment techniques can be used either for training alignment or for testing alignment, but not both, and that some techniques should be kept for testing only. The author extends the machine learning principle of held-out test sets to alignment, arguing that training with all monitoring methods would make a model appear aligned to every evaluation method while losing the ability to verify the model's behavior. The post lists chain-of-thought monitoring, human feedback, probes, honeypots, automated auditing, and other techniques that can be used in both directions, and states that chain-of-thought monitorability is the only known alignment technique that is genuinely held out, and that it is dying out. The author takes as an example a paper from the Anthropic MATS school proposing the gradient routing method GRAFT, and argues that the Forget adapter responsible for reward hacking could be retained, observing at inference time whether the model routes computation through it, in order to catch improper model behavior. The author concludes by suggesting that frontier AI companies disclose which alignment techniques are held out for testing, expand the total number of techniques, and reserve most of them for testing purposes only.

Read originallesswrong.com