SFT Lessons Shared Among Alignment, Model Organisms, and Toy Models

The researchers propose transplanting supervised fine-tuning (SFT) experience among three domains—alignment training, model organisms, and toy models—and test its transferability with three sets of experiments. The first set transplants behavior generalization experience from alignment training to toy models, finding that training on the reasons behind behavior (as in the Teaching Claude Why approach) generalizes behavior better than training only on behavior examples. The second set transplants capability preservation experience from model organisms to the Model-Spec Midtraining alignment setting, finding that performing SFT on outputs not written by the student model harms capabilities, and that mixing in benign data from the same model and same policy avoids most of the harm while still embedding the target behavior. The third set transplants robustness experience from model organisms to the same setting, finding that subsequent benign SFT can erase the alignment behavior while preserving capabilities, indicating that capability preservation alone is insufficient to guarantee robustness to subsequent training. The authors argue that transplanting SFT experience across domains can benefit all domains and suggest that researchers borrow techniques from outside their own field.