Skip to content

Alignment via Reflect-Reasoning: Learning How to Follow Principles

Alignment with Reflect-Reasoning: Learning How To Follow Principles

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2611.17398· Henry Bell, Caroline Zhang, Dhaval Potdar, Mohammed Mobasserul Haque, Brandon FainDuke UniversityIndexed by this site · 4 days ago· Originally publishedAlignment· Its place in the scan record →

Translation in progress — untranslated text is omitted.

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

AI summaryResearchers propose the Reflect-Reasoning training method, which teaches models to learn how to reason around alignment principles rather than simply specifying which principles to follow. Addressing the problem that alignment specifications depend on cultural context and deployment scenarios and that most alignment techniques fix principles at training time, the method consistently outperforms system prompts and explicit prompting strategies for in-context reflective reasoning across five models, and generalizes to out-of-distribution medical and financial principle domains. The authors also report that the method reduces rare but severe principle violations, does not weaken the model's existing safety engineering, withstands jailbreak attempts using adversarial constitutions, and enables models to adapt to new principles without additional fine-tuning and to produce transparent reasoning traces.

Questions
Alignment principles often vary with culture and deployment scenario, but most alignment techniques fix the principles at training time; deployers actually rely on system prompts, which the authors consider insufficient for substantive adherence to subjective principles.
Methods
Proposes Reflect-Reasoning: first SFT on reflective reasoning traces generated by gpt-4.1, then reinforcement with GRPO, with a reward combining format rules, a length penalty, and two sets of pairwise comparisons by an LLM judge; during training, prompts are randomly paired with principles so the model learns to perform "generate—reflect—revise" on new principles given at inference time.
Experiments and results
The authors report: on held-out principles, the five models achieve win rates of 82.5–94.7% against the system prompt and 66.8–86.9% against the explicit reflection prompt, and generalize to the medical and finance domains; the rate of severe principle violations decreases; HarmBench ASR is similar to the base model.
Limitations and future work
The authors note that the evaluation relies on an LLM judge, and that both the judge and the human annotators are in the United States; training covers only 7–32B models and English; the reasoning traces may be unfaithful. The safety analysis is done only on Tulu-3-8B-SFT, and the CAI comparison is done only on Olmo-3-7B.

Related papers

Read originalarxiv.org