OpenAI research: Reinforcement learning toward broadly and lastingly beneficial models
Reinforcement Learning Towards Broadly and Persistently Beneficial Models
Quick read
Based on the paper's full text (AI-generated); refer to the original
OpenAI researchers trained a model with reinforcement learning on beneficial traits; it exceeded a compute-matched baseline on 44 of 53 independently constructed out-of-distribution alignment and beneficiality evaluations (83.0%), with an average improvement of 9.1 percentage points and significant regression on only 3. The training mixture was 5% beneficial-trait data and 95% standard RL data, with the dataset covering 12 domains including health, education, and law, and targeting 15 beneficial traits such as honesty, corrigibility, and risk awareness. When trained only on beneficial data from the health domain, the model improved on 17 of 19 non-health evaluations, with coding reward hacking improving by 26.4 percentage points and chain-of-thought deception by 6.8 percentage points; conversely, after excluding all health and science data, health and mental health evaluations still improved. The model degraded less under harmful persona prompts and harmful fine-tuning; a generic helpfulness reward could not reproduce the gains, and improvements remained after excluding refusal samples, while capability evaluations such as GPQA Diamond and SWE-Bench Pro did not regress.
- AI system deployment scenarios are increasingly diverse, and model alignment needs to generalize beyond the training distribution; prior research has found that harmful training in a narrow domain can trigger broad emergent misalignment, and the authors want to test the opposite direction—whether training on beneficial behavior can bring broad and lasting alignment generalization.
- Constructs a synthetic dialogue dataset covering 12 domains such as health, law, and education and 15 beneficial traits (such as truthfulness, fairness, risk awareness, corrigibility), trains models with RL, and evaluates on more than 50 independent out-of-distribution alignment and beneficiality benchmarks; also conducts control experiments such as health-domain-only, excluding the health domain, and generic helpfulness reward, as well as persistence tests under adversarial prompts and harmful finetuning.
- Compared with a compute-matched baseline, beneficial-trait RL is better on 44 of 53 out-of-distribution evaluations (83%), with an average improvement of about 9 percentage points; training with only health-domain data also improves reward hacking, deception, and misalignment evaluations in non-health domains; the model is more resistant to harmful persona prompts and harmful finetuning while retaining responsiveness to beneficial guidance, and no decline in capability or monitorability was observed.
- The authors acknowledge that some evaluations may share latent behavioral features with the trained traits, so the degree of "true out-of-distribution" is limited; the persistence experiments used the pre-RL baseline and cannot isolate whether this effect is specific to beneficial-trait RL; the refusal rate also rises slightly in ordinary conversation; the conclusions still require validation with more models, finetuning objectives and evaluation suites.
Using 53 out-of-distribution evaluations and controlled experiments that exclude the training domain, the paper shows that beneficial-trait RL can yield cross-domain alignment generalization, and the capability evaluations tested did not regress.