OpenAI proposes Deliberative Alignment: reasoning makes language models safer

1. 01What problem does this paper attempt to solve?

   The authors aim to address the problem that models lack reasoning and rely on implicit learning, making them vulnerable to adversarial jailbreaks and often over-refusing.

   Large language models struggle to balance adversarial attack robustness and benign request compliance in safety scenarios, exhibiting the dual defect of being susceptible to jailbreak induction and over-refusal.

   • Limitations of existing safety training mechanisms: The authors argue that modern models mostly rely on SFT and RLHF, responding immediately with fixed computation at inference time without deliberating in complex safety scenarios; moreover, models mainly induce safety standards implicitly from annotated data, lacking direct learning of explicit safety specifications, resulting in low data efficiency and difficult generalization.
   • Bottleneck in scaling annotated data: As model capabilities improve, the pool of qualified human annotators shrinks, and the traditional fine-tuning paradigm that relies on large-scale human-annotated data faces obstacles in scalability.
   • Core observation and hypothesis: The authors observe that if a reasoning model is directly provided with safety policies, the model can usually correctly deduce how to respond to potentially unsafe prompts; allocating test-time computation through chain-of-thought (CoT) reasoning enables the model to identify the policy and generate safe responses.
   • Solution proposed in the paper: The paper proposes Deliberative Alignment, which directly teaches safety specifications to the model during training, prompting it to explicitly retrieve and reason over policy specifications in the chain of thought before generating the final answer, without requiring human-written chains of thought or answers.

   Collapse this answer

2. 02What related research exists?

   Related work includes safety training such as RLHF and CAI as well as inference-time correction; the authors position this paper as directly teaching specifications and reasoning over them in the chain of thought.

   • Safety training based on human and AI feedback: SFT and RLHF (Ouyang et al., 2022; Christiano et al., 2017) inject safe behavior through instruction fine-tuning and human-preference reinforcement learning; DPO (Rafailov et al., 2024) skips the reward model and directly optimizes the policy using preference data. The authors note that in these methods the specifications are only used to guide annotators, and the policy model itself never explicitly receives or optimizes the specification content.
   • Principle-based alignment methods: Constitutional AI (CAI) (Bai et al., 2022) uses preset principles to guide the model in self-critique and revision during the SFT stage, and trains a reward model with AI preferences. The authors note that CAI only uses the revised final answer for training, the critique process does not participate in optimization, and specification knowledge is not retained for the model at inference time.
   • Inference-time revision and safe generation: Self-REFINE (Madaan et al., 2023) generates feedback through few-shot prompting and iteratively revises outputs. The authors note that such methods rely on preset language model programs or predetermined reasoning paths, whereas this paper uses the built-in chain of thought for automatic reasoning; Backtracking (Zhang et al., 2024) trains the model to generate a reset token and regenerate when it detects partially unsafe outputs, and the authors note that it is limited to a single reset per response and does not directly teach the model safety specifications.
   • Baseline methods and benchmarks: Baseline models include GPT-4o, Gemini 1.5 Pro, Gemini 1.5 Flash, Claude 3.5 Haiku, Claude 3.5 Sonnet, as well as an inference-time specification baseline where the full specification is included in the prompt; evaluation benchmarks include StrongREJECT, XSTest, WildChat, SimpleQA, BBQ, and internal violation-content and style evaluation sets.
   • Positioning of this paper: The authors state that deliberative alignment is a method that directly teaches the model the text of safety specifications and trains it to reason over the learned specifications through chain of thought at inference time to give safe answers.

   Collapse this answer

3. 03How does the paper address this problem?

   In the SFT stage, context distillation is used to generate chain-of-thought with specification references and filter them; in the RL stage, the chain of thought is hidden and a discriminator model provides rewards.

   The overall approach is to use two-stage training to make the model actively recall and reason about safety specifications in its chain of thought, proposing the Deliberative Alignment framework.

   Problem Setup and Formalization

   The goal is to train a generative reasoning model G_spec that satisfies safety specifications, starting from an initial reasoning model G_base. The safety specifications include policies for different categories of violating content and response style guidelines (hard refusal, self-harm safe completion, compliant suggestion safe completion, etc.). To control context length, a specific specification spec(category) is constructed for each category, retaining global high-level information and providing detailed rules only for the relevant category.

   Supervised Fine-Tuning (SFT) Stage and Data Generation

   1. Context Distillation Generation: For prompts with safety categories, the corresponding category specification and guidance instructions are added to the system prompt, and G_base, which has not been trained on safety data, generates chain-of-thought and responses that include specification references.
   2. Automated Quality Filtering: A reasoning discriminator model G_RM that also knows the specifications is used to score candidate samples k times independently, taking the minimum of the k scores as the final score, and only high-scoring samples are retained; when noisy label metadata is present, G_RM is prompted to reference other AI judgments and provide reasons.
   3. Specification Removal Fine-Tuning: The safety specification text is stripped from the prompts, leaving only the original prompt, chain of thought, and output as triples to train G_base, teaching the model to autonomously recall and reason when the specification is not provided.

   Reinforcement Learning (RL) Stage

   In the RL stage, for safety-related prompts, the discriminator model G_RM, which knows the safety specifications, provides reward signals.

   • Hidden Chain-of-Thought Mechanism: Unlike SFT data filtering, the chain of thought is hidden from G_RM during RL training to avoid directly applying optimization pressure on the chain of thought, reducing the risk of inducing deceptive chain-of-thought.

   Collapse this answer

4. 04What experiments did the paper conduct?

   In the experiments, o1 outperforms GPT-4o in both jailbreak prevention and false-refusal prevention, and ablations show that SFT and RL each make independent contributions.

   Experimental Setup

   • Models tested: o1, o1-preview, o3-mini, and reduced training versions of o1-mini variants used in ablation studies.
   • Baseline models: GPT-4o, Gemini 1.5 Pro, Gemini 1.5 Flash, Claude 3.5 Sonnet, Claude 3.5 Haiku, and a baseline with specification provided at inference time in ablations.
   • Datasets and benchmarks: Challenging Refusal Evaluation (unsafe requests from production traffic), WildChat (top 200 samples per category from ModAPI out of 1 million public conversations), StrongREJECT (60 forbidden prompts each with 35 jailbreak techniques, totaling 2100 samples), XSTest, Transformation Exception dataset, Policy Overrefusals, SimpleQA (4000 short factual questions), BBQ (bias evaluation across 9 social dimensions).
   • Main metrics: not_unsafe (non-unsafe rate), style_adherence (style adherence rate), [email protected] (safety under top 10% attacks per prompt on StrongREJECT), not_overrefuse (non-overrefusal accuracy), accuracy, 1 - Hallucination Rate, P(not stereotyping | ambiguous, not 'unknown'), Policy Retrieval Accuracy.
   • Evaluation method: Automated evaluation uses an autograder that knows the policies; StrongREJECT additionally uses 35 trained reviewers for human evaluation with a 1-3-5 scoring system (two independent scores plus expert arbitration); error bars are computed with 1000 or 10000 bootstrap resamples.

   Main Results

   The table below presents the evaluation results of each model on core safety metrics:

   Swipe horizontally to read the table

   Model	Violating Content (Challenging Refusal)	Jailbreak Defense (StrongREJECT)	Overrefusal (XSTest)	Transformation Exception	Safe Completion (Self-harm)
   GPT-4o	0.8	0.37	0.88	0.28	0.04
   o1-preview	0.934	0.66	0.976	0.72	0.01
   o1	0.9	0.88	0.93	0.89	0.92
   o3-mini	0.94	0.75	0.9	0.89	0.87

   According to Table 1. Note: The table omits metrics such as WildChat, hard refusal style, compliant suggestion style, and internal policy overrefusals.

   • Breakthrough in the trade-off between jailbreak resistance and overrefusal avoidance: The authors state that the o1 series advances the Pareto frontier of jailbreak resistance and overrefusal avoidance, achieving a [email protected] of 0.88 on StrongREJECT while maintaining a high non-overrefusal rate of 0.93 on XSTest.
   • Version differences in safe completion style: o1-preview has only 0.01 on self-harm safe completion; the authors explain this is due to an update in the safe completion guidelines between the releases of o1-preview and o1, and o1-preview is evaluated against the new standard it was not trained on; o1 improved to 0.92 after introducing the discriminator model in RL.
   • Benign boundary handling capability: On the transformation exception task, o1's not_overrefuse rate reaches 0.89, an improvement over GPT-4o's 0.28, demonstrating the ability to accurately follow benign transformation exceptions.

   Comparison with External Models and Human Evaluation

   • What was tested: Comparison with external models on public benchmarks (Figure 12), and human review with 35 people on StrongREJECT (Table 4).
   • Results: In the StrongREJECT human evaluation, o1's [email protected] reaches 0.92 (average score 1.020), Claude 3.5 Sonnet is 0.90 (average score 1.047), GPT-4o is 0.35 (average score 1.335), and Gemini 1.5 Pro is 0.08 (average score 1.667) (Table 4). On XSTest, o1 is 0.93, only behind Gemini 1.5 Flash (0.94), but the latter has only 0.05 on StrongREJECT (Figure 12).
   • Authors' interpretation: The authors state that the human evaluation results confirm the consistency of the autograder evaluation, and o1 achieves leading performance in balancing adversarial jailbreak defense and benign compliance.

   Analysis of Inference Compute and Out-of-Distribution Generalization

   • What was tested: The impact of test-time chain-of-thought reasoning compute on safety performance (Figure 13), and out-of-distribution generalization on non-English and coding data (Table 3).
   • Results: Increasing reasoning compute significantly improves performance on StrongREJECT and compliant suggestion safe completion, while other evaluations are relatively flat (Figure 13); on models without coding and non-English safety data, coding jailbreak accuracy reaches 0.97±0.02 (full data: 0.95±0.03, no safety baseline: 0.65±0.06), and multilingual jailbreak accuracy reaches 0.69±0.01 (full data: 0.68±0.01, no safety baseline: 0.44±0.01) (Table 3).
   • Authors' interpretation: The authors believe that reasoning compute helps handle complex safety scenarios such as composite jailbreaks; the model's performance on unseen coding and languages is comparable to full data, indicating that deliberative alignment exhibits out-of-distribution generalization.

   Other Ablations and Analyses

   • Training stage ablation (Figure 14, o1-mini variants): When both SFT and RL include safety data, StrongREJECT reaches 0.87, Hard Refusal 0.88, XSTest 0.91; SFT only: 0.81, 0.85, 0.93; RL only: 0.73, 0.81, 0.94; no safety training: 0.44, 0.13, 0.98.
   • Inference-time specification baseline (Figure 14): When the full specification is provided directly in the prompt, StrongREJECT is 0.80, Hard Refusal is 0.54, and self-harm safe completion is only 0.09, all inferior to models trained with deliberative alignment.
   • Policy retrieval accuracy (Table 2): Models trained on full data achieve retrieval accuracy for correct policy categories of 0.75, 0.91, and 0.54 under Hard Refusal, Safe Completion, and Comply, respectively, compared to 0.27, 0.21, and 0.09 for the no-safety baseline.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that future work needs to address the evolution of safety specifications and the risk of deceptive chain-of-thought, and they plan to continue exploring more robust methods for chain-of-thought supervision and monitoring.

   Limitations and future directions noted by the authors

   • Adaptation to specification evolution: The authors expect OpenAI's policies to keep evolving, and models need the ability to adapt to any future policy requirements (Section 6).
   • Goal misalignment and power-seeking risks: The authors note that as model capabilities increase, models may develop goals that deviate from human intent, or even pursue instrumental goals such as self-preservation or resource acquisition (Section 6).
   • Monitoring deceptive chain of thought: The authors note that future work should actively invest in alignment research such as monitoring deceptive chain of thought to ensure more powerful systems remain aligned (Section 6).
   • Optimization pressure from reinforcement learning on chain of thought: The authors note that hiding the chain of thought from the discriminator model during the RL stage is intended to avoid directly applying optimization pressure that could foster deceptive chain of thought, which limits direct supervision of the chain of thought in RL (Section 2.4).
   • Increased false refusals in specific settings: The authors note that in the reduced-scale o1-mini ablation setting, safety training increased false refusals (Section 4.1).
   • Factual hallucination rate performance: The authors note that on SimpleQA, although o1 achieves 0.47 accuracy, its hallucination rate is higher than that of the two Claude models tested (Section 3.2).

   Experimental coverage

   • The ablation experiments and mechanism analyses (Sections 4.1, 4.2, 4.3) are all conducted on a reduced-scale o1-mini variant, and no corresponding ablation data are reported on the full-scale o1 model.
   • External model comparisons mainly cover Gemini 1.5 Pro, Gemini 1.5 Flash, Claude 3.5 Sonnet, Claude 3.5 Haiku, and GPT-4o (Section 3.2).
   • Out-of-distribution robustness experiments only test jailbreak scenarios based on coding (250 forbidden prompts) and multilingual translation (Section 4.3 and Table 3).
   • The paper reports agreement between automatic evaluation and human evaluation on StrongREJECT, but does not report human agreement metrics for internal evaluation sets (Challenging Refusal, Transformation Exception, etc.).
   • The paper does not report the specific number of queries and computational time consumed by the discriminator model GRM at each training stage.

   Collapse this answer

6. 06Summarize the main content of the paper

   Deliberative alignment, by having the model reason about safety specifications in its chain of thought, achieves high-precision policy alignment while balancing jailbreak robustness and benign compliance.

   Deliberative alignment is a safety alignment paradigm that directly teaches safety specifications to the model and uses test-time chain-of-thought reasoning to explicitly reason and generate compliant responses.

   • Problem background: Traditional SFT and RLHF rely on implicit pattern matching and lack test-time deliberation, causing models to be insufficiently robust when facing complex boundaries or adversarial jailbreaks, while also being prone to false refusals on benign requests.
   • Method design: In the first stage, context distillation makes the base model generate chain-of-thought and answers while carrying the specifications; after filtering by a discriminator model, the specifications are removed and SFT training is performed. In the second stage, a discriminator model that has mastered the specifications provides RL rewards for outputs, and the chain of thought is hidden during RL to avoid inducing deceptive reasoning.
   • Main experimental results:
     1. On the StrongREJECT jailbreak benchmark, o1 achieves a [email protected] of 0.88, and 0.92 under human review, higher than GPT-4o's 0.37 and 0.35 (Table 1 and Table 4).
     2. On the XSTest false-refusal test, o1 achieves a not-overrefuse rate of 0.93, and 0.89 on the complex transformation exception task, outperforming GPT-4o's 0.28 (Table 1).
     3. On the self-harm safety completion task, o1's style adherence reaches 0.92, higher than GPT-4o's 0.04 (Table 1).
     4. In out-of-distribution generalization experiments, when never trained on coding safety data, o1-mini achieves an accuracy of 0.97±0.02 on coding jailbreak evaluations (Table 3).
   • Conclusions and implications: The authors believe chain-of-thought reasoning can convert test-time computation into safety gains; the purely synthetic data pipeline removes dependence on large-scale human annotation, but future work must remain vigilant about the risk of deceptive chain-of-thought in more capable models.

   Collapse this answer

Related papers