Constitutional Classifiers: defending against universal jailbreaks after thousands of hours of red-teaming

The core problem is how to build a language model safety defense mechanism that can resist universal jailbreak attacks while being practically deployable (low latency, low false positives, and reasonable overhead).

• Scenario and importance: As language models improve in scientific domains such as chemistry, biology, radiological, or nuclear (CBRN), malicious users may use jailbreak prompts to induce the model to provide detailed instructions for high-risk processes; the authors believe that if non-experts can reliably obtain specific operational information, this poses a non-expert uplift risk.
• Shortcomings of existing methods: The authors state that single-model defenses relying solely on harmlessness fine-tuning are easily bypassed by universal jailbreak strategies such as many-shot jailbreaking or "God-Mode", and fine-tuning on red-team attack samples is difficult to generalize across domains; meanwhile, external guardrails often suffer from excessive false-positive refusals or disrupt the token-by-token streaming experience, making them difficult to deploy in practice.
• Core observation and hypothesis: A single classifier in isolation may not achieve complete invulnerability, but an input classifier and a streaming output classifier working together can form a "Swiss cheese" multi-layer defense-in-depth; furthermore, by explicitly defining a natural language constitution of allowed and restricted content, it is possible to generate diverse synthetic data at scale and establish clearer classification boundaries.
• Threat model:
  • Attacker goal: Reliably bypass safety mechanisms across most queries in specific high-risk domains (e.g., CBRN), inducing the system to leak highly detailed, actionable, specific harmful information.
  • Attacker knowledge: The attacker has black-box interaction and API access, knows the system's defense architecture, but does not have white-box access to the model's internal weights or gradients.
  • Attacker capabilities: Can use manual or automated tools to construct complex multi-turn dialogues, role-playing, text transformations (e.g., Base64 encoding, multilingual translation), benign paraphrasing, and length exploitation attack payloads.
  • Victim system: A frontier large language model system providing interactive API and text generation services (e.g., Claude 3.5 Sonnet).
• Proposed solution in the paper: Proposes a Constitutional Classifiers framework that generates synthetic data from explicit natural language constitutions for fine-tuning, building a dual defense system consisting of an input classifier and a streaming output classifier supporting token-by-token prediction.

Classifier guards

• Markov et al. (2023): Studied the role of data quality and active learning in training content moderation APIs, and trained classifiers using synthetic data.
• Inan et al. (2023), Rebedea et al. (2023), and Chi et al. (2024): Developed input-output guardrail systems such as Llama Guard, NeMo Guardrails, and Llama Guard 3 Vision, respectively; Kim et al. (2024) proposed that output filtering can mitigate a variety of prevalent jailbreak attacks.
• Wang et al. (2024): Studied classifier guards in the specific narrow domain of bomb-making; the authors state their work shows that additional classifier fine-tuning can improve performance.

Finetuning on red-teaming attacks

• Ganguli et al. (2022) and Ouyang et al. (2022): Identified harmful behaviors through human red-teaming and fine-tuned models based on human feedback to eliminate harmful tendencies.
• Perez et al. (2022), Chao et al. (2024), and Mehrabi et al. (2024): Used language models for automated red-teaming (e.g., FLIRT); the authors note that frontier models trained with such fine-tuning remain vulnerable to universal jailbreaks (e.g., many-shot jailbreaking, GCG), and in preliminary experiments found that such fine-tuning is difficult to generalize reliably across modalities or from text to code.

Model-internals approaches & Unlearning

• Alain and Bengio (2016), Zou et al. (2024), and Casper et al. (2024): Used methods that rely on model internal representations, such as linear probes, circuit breakers, or latent adversarial training, to defend against jailbreaks; the authors note that constitutional classifiers do not require changes to the LLM fine-tuning pipeline and are more flexible in practical deployment.
• Li et al. (2024), Zhang et al. (2024), and Lynch et al. (2024): Eliminated dangerous knowledge within the model through machine unlearning, or intercepted via pretraining data filtering; the authors note that machine unlearning often fails to completely erase target knowledge, and data filtering lacks flexibility.
• Zaremba et al. (2024): Explored enhancing adversarial robustness by scaling inference-time compute; the authors note this approach introduces high latency and inference overhead.

Comparison baselines and evaluation benchmarks

• Baseline methods: A model fine-tuned for harmlessness alone without an external classifier (Claude 3.5 Sonnet HHH model), a Helpful-only model, a 0-shot prompt classifier based on Claude 3.5 Sonnet, a CoT chain-of-thought prompt classifier, and a 32-shot prompt classifier.
• Evaluation benchmarks/datasets: HackerOne crowdsourced red-teaming dataset, 5,681 chemical weapons held-out composite jailbreak dataset, Wildchat conversation set, OR-Bench false refusal benchmark, random sample of Claude.ai production traffic, and chemistry subsets of GPQA and MMLU.

Differentiation from existing work

• The authors state that constitutional classifiers are professionally fine-tuned using synthetic data generated from natural language constitutions, allowing rapid adaptation to new threats without human annotation, and through the combination of an input classifier and a streaming Value Head output classifier, demonstrate defense against universal jailbreaks under thousands of hours of intensive human red-teaming, while maintaining extremely low production false refusal rates and streaming inference efficiency.

The overall approach is to build a Constitutional Classifiers framework, using natural language rules to define harmful and harmless boundaries and generate high-quality synthetic training data, deploying input classifiers and streaming output classifiers to form layered defense.

Constitutional specification design and synthetic data generation

• Constitution structure: Harmful and harmless constitution rules are written in natural language. The harmful constitution covers specific technical aspects of chemical weapons acquisition, production, formulation, weaponization, and delivery; the harmless constitution details harmless chemistry (e.g., everyday cleaning, high school chemistry experiments), non-chemical harmful topics (e.g., routine cyberattacks, ordinary crime), and non-chemical harmless topics.
• Synthetic data generation process: A helpful-only model is used as the generator, prompted by constitution categories to generate a large number of targeted queries and corresponding harmful and harmless responses; then a prompt-driven language model filters out accidental refusal content in synthetic samples.
• Multi-dimensional data augmentation: Synthetic samples undergo multilingual translation, model rewriting, system prompt variations, and semantic-preserving transformations combining known jailbreak patterns to expand training data coverage and representational diversity.

Automated red teaming and data filtering

• Two-stage attack generation: Automated red teaming (ART) techniques are used, first prompting the model to generate attack outlines covering multi-turn dialogues, then resampling and filling in specific interactions stage by stage to form long-context and multi-turn jailbreak attacks.
• Sample cleaning and distribution balancing: Claude 3.5 Sonnet is used to remove low-quality attack samples that fail to induce harmful responses based on the rating rubric; simultaneously, general benign dialogues on harmful topics are automatically generated to ensure ART samples account for less than 2% of the final training set, avoiding training noise.

Input and output classifier training and streaming optimization

• Input classifier formalization: Input detection is modeled as a next-token prediction task, using a fine-tuned language model to directly determine whether user input contains high-risk intent, and specifically training it to recognize encoded communication or obfuscated payloads.
• Output classifier architecture: A linear Value Head is added on top of the backbone language model, predicting the full-sequence harmfulness label y when generating prefix sequence t_1:T′, jointly optimizing the next-token prediction loss and binary cross-entropy loss: L(y, t_1:T) = λ · L_NTP(t_1:T) + ∑_T′=1^T L_BCE(y, p(y = 1 | t_1:T′)), where λ is a regularization hyperparameter.
• Dynamic interpolation of prediction target: To balance optimization stability of single-token gradients with the fit of cumulative maximum in streaming deployment, a weight ω is linearly increased from 0 to 1 during training (completed in the first 3/4 of training): p(y = 1 | t_1:T′) = (1 − ω) · σ(z(t_1:T′)) + ω · max_{τ ≤ T′} σ(z(t_1:τ)).

Threshold calibration and streaming truncation mechanism

• Constrained threshold search: Under false positive upper bounds set by multi-turn Wildchat (incremental FPR ≤ 0.5%), single-turn Wildchat (FPR ≤ 5%), and OR-Bench (incremental FPR ≤ 5%), grid search determines classification thresholds.
• Streaming real-time intervention: During token-by-token decoding, the output classifier uses cached intermediate activations to compute harmfulness probability synchronously; once the prediction at a token exceeds the preset threshold, the system immediately truncates generation and stops streaming output.

Experimental Setup

• Models under test: Claude 3.5 Sonnet (target model for red-team crowdsourcing system and base model for prototype classifier), Claude 3.5 Sonnet (New) (target model for automated evaluation), Claude 3.5 Haiku (base model for automated evaluation classifier fine-tuning).
• Datasets and scale: CBRN red-team crowdsourcing set (10 non-informational harm target queries), chemical weapons held-out jailbreak set (5,681 composite jailbreak samples with unseen primitives and transformations), random sample of Claude.ai production traffic (5,000 real conversations), Wildchat conversation set (single-turn and multi-turn, plus 1,000 chemistry subset), OR-Bench, GPQA-Chemistry, and MMLU-Chemistry.
• Comparison baselines: A model fine-tuned for harmlessness alone without an external classifier (HHH model), Helpful-only model, 0-shot prompt classifier based on Claude 3.5 Sonnet, CoT prompt classifier, and 32-shot prompt classifier.
• Main metrics: Attack success rate (ASR / jailbreak success rate), refusal rate (Refusal Rate / false positive rate FPR), true positive rate (TPR), relative inference cost overhead (Inference Cost).
• Review and judgment method: Multi-stage LLM scoring rubric pipeline; in the red-team phase, rubrics are constructed by comparing model responses to helpful-only responses, and a score reaching 50% of the helpful-only mean is judged as a jailbreak; in the automated evaluation phase, an LLM judges whether the response covers at least two harmful topics with coverage reaching 25% of the helpful-only mean (and does not trigger a refusal judgment).
• Sample size and repetitions: Red team received 113 reports (85 valid for analysis); automated held-out jailbreak set 5,681; inference cost evaluation sampled 1,000 production input-output pairs; streaming output evaluation sampled 10,000 jailbreak attack outputs; the paper does not report the number of repetitions with multiple random seeds.

Main Results

Swipe horizontally to read the table

(According to Figure 6 and Section 5.2; prompt baselines are shown only as bar chart ranges in Figure 6, and the main text does not provide exact textual numbers)

• Robustness improvement: The authors state that the constitutional classifier reduced the held-out jailbreak attack success rate against Claude 3.5 Sonnet (New) from 86% in the no-classifier baseline to below 5%, intercepting over 95% of held-out attack attempts, and no single text transformation was able to break all 10 test queries.
• Deployment cost control: The authors note that the improvement in defense performance comes with a small practical cost: in a 5,000-conversation sample of real user production traffic on Claude.ai, the absolute refusal rate increased by only 0.38%, the lowest increase among all baselines.
• Computational efficiency maintained: The authors report that by using a smaller fine-tuned Claude 3.5 Haiku and caching mechanisms, the additional inference compute overhead is only 23.7% compared to the base model, lower than the prompt classifier baselines.

Human Red-Team Crowdsourcing Results Analysis

• Test setup: A bug bounty program targeting the CBRN domain was conducted on the HackerOne platform, inviting 405 participants (estimated 183 active red-teamers, totaling approximately 4,720 hours) to attack and test the Claude 3.5 Sonnet prototype classifier-guarded system, with 10 non-informational harm target questions.
• Test results: Of the 113 submitted reports (85 valid for analysis), no report achieved half the helpful-only level of detail on all 10 questions; under a stricter standard requiring detail comparable to helpful-only, no red-teamer answered more than 6 questions across all submissions (Figure 4B); success rates varied significantly across questions, with the hardest question having only 4 reports reaching half detail and the easiest over 30 (Figure 4C).
• Authors' interpretation: The authors state that red-teamers failed to find a universal jailbreak that could reliably extract detailed information across most target queries; high-scoring reports mainly exploited benign paraphrasing and length exploitation to game the scoring rubric, rather than truly bypassing the defense mechanism.

Defense Component Ablation and Streaming Truncation Evaluation

• Test setup: The contribution of each component to ASR was evaluated on a subset of red-team attack samples, and the effectiveness of the streaming output classifier in blocking harmful information in real-time at the token level was evaluated on 10,000 sampled jailbreak attack outputs.
• Test results: The HHH model alone had an ASR of approximately 14% (helpful-only approximately 16%), which dropped to 2% with the input classifier, and to 0.5% with both input and output classifiers (Figure 5); across 10K jailbreak attacks, the system blocked 96.70% overall, and among truncated outputs intercepted by the output classifier, only 117 were judged to have leaked harmful information before truncation, i.e., a streaming failure-to-block rate of only 1.17%.
• Authors' interpretation: The authors state that the combination of input and output classifiers provides much stronger defense-in-depth than harmlessness training alone; the vast majority of attacks are intercepted by the input classifier or blocked early in token generation by the output classifier, effectively preventing exposure of harmful information to users.

Scaling and Data Augmentation Trends

• Test setup: Analyzed the impact of classifier model size (5 sizes in the Claude 3 Sonnet family), training data volume (relative token scale from 2⁻⁶ to 2⁰), and number of data augmentation transformations (1 to 66) on true positive rate (TPR) under a fixed false positive constraint.
• Test results: Classifier performance improves with increasing model size, training data volume, and number of transformations (Figure 7); the generalization gap of the input classifier narrows as model size increases; adding harmless constitution data improves robustness by over 40% (Appendix E.5, Figure 13).
• Authors' interpretation: The authors believe classifier robustness scales well, and that increasing training data and data augmentation can effectively improve generalization to unseen jailbreak patterns.

Other Ablations and Analyses

• Classification task modeling format: The next-token prediction architecture outperforms the Value Head architecture in TPR for both input and output classification (Figure 9).
• Base model initialization choice: Classifiers initialized from the Claude 3.5 Haiku base model outperform those initialized from HHH fine-tuning and Helpful-only fine-tuning (Figure 12).
• Harmless constitution category ablation: Removing any of the harmless chemistry, harmless non-chemistry, or harmful non-chemistry subcategories results in a decrease in classifier TPR (Figure 14).
• Few-shot prompt scaling: Increasing prompt examples from 2 to 32 improves classifier TPR, but context grows linearly and still does not reach the performance of the fine-tuned small classifier (Figure 11).
• Specialized domain false refusal performance: False refusal rate is 1.50% on MMLU-Chemistry, while reaching 26.05% on GPQA-Chemistry (Appendix E.2).
• Chemistry user query false refusal performance: On the 1,000 Wildchat chemistry subset, the input and output classifiers add an additional 3.4% refusal rate (Appendix E.8).

Negative Results and Exceptions

• High refusal rate of prototype system: The early red-team prototype system, due to overemphasis on robustness, resulted in a refusal rate of approximately 44% on production Claude.ai traffic (Section 4.1).
• High false refusal on GPQA professional chemistry questions: False refusal rate reaches 26.05% on GPQA-Chemistry professional questions; the authors note that precise classification of high-difficulty technical content requires complex domain understanding beyond the current classifier's capabilities (Appendix E.2).
• Noise in automated red-team benign data: ART data was not included in the chemical weapons classifier training because preliminary experiments showed that ART-generated "benign" samples were too close to constitutional harmful categories, introducing excessive noise into the training set (Appendix D.2).
• Deployment infrastructure flaw: During red-teaming, a report appeared to achieve a universal jailbreak, but subsequent investigation revealed it was not a failure of the classifier itself, but rather an error in the deployment engineering system that still output up to 128 tokens to the user after the classifier flagged content as harmful (Section 3, footnote 2, and Section 4.2, footnote 1).

Limitations and future directions noted by the authors

• Red team representativeness and incentive limitations: The authors note that the red team participants represent only a subset of potential adversaries, and new attacks will emerge as adversarial methods evolve; moreover, participants receive bonuses upon reaching a 50% detail threshold, and with stronger incentives they might explore more destructive attacks (Section 4.2 Limitations).
• Rating rubric difficulty in discerning practical harm: The authors note that the keyword-based rubric evaluation mechanism struggles to effectively distinguish harmless common knowledge from specific actionable information that truly enhances capability, which may lead to both false positives and false negatives (Section 4.2 Limitations).
• Test environment limited to single chat interaction: The authors note that experiments mainly focus on chatbot scenarios with fixed queries, and the robustness of classifiers in other real deployment contexts (such as complex agent workflows, long-context tool calls) remains an open question to be validated (Section 4.2 Limitations).
• Potential system vulnerabilities still require complementary defenses: The authors note that experience shows system vulnerabilities are likely to resurface as testing continues, and classifiers alone are insufficient for long-term assurance; responsible deployment must be supplemented with multi-layer defenses such as rapid vulnerability hotfixes and anomaly jailbreak monitoring (Section 7 Conclusion).
• Simplified assumptions in capability enhancement estimation model: The authors note that their model for estimating capability enhancement in multi-step scientific processes assumes each step is fully independent and of uniform difficulty, representing a highly simplified illustrative estimate that does not cover feedback loops and non-expert trial-and-error recovery mechanisms in real complex processes (Appendix C).
• Risk of subjective bias in red team hour estimation: The authors note that the calculation model for total red team hours relies on calibration using self-reported time data from red team members in questionnaires, which may have systematic overestimation (Section 4.1 and Appendix H.4).

Experimental coverage

• Models tested: Experiments only evaluated Anthropic's Claude 3.5 Sonnet, Claude 3.5 Sonnet (New), and Claude 3.5 Haiku models, not including other model families.
• Harm domain coverage: Experiments and constitutional rules only cover the CBRN domain (focusing on chemical weapons and specific biological harm-related queries), and the paper explicitly states that classifiers were not trained to recognize other harmful categories (e.g., robbery, cyberattacks).
• Test query scale: Human red team evaluation used 10 preset non-information harm target CBRN queries, and automated held-out jailbreak evaluation also targeted 10 preset chemical weapons queries.
• Data and sample counts: The automated held-out jailbreak set contains 5,681 attacks; production traffic false refusal evaluation sampled 5,000 conversations from Claude.ai; the Wildchat chemical subset contains 1,000 conversations.
• Unreported information: The paper does not report random seeds for model sampling in automated evaluation or variance across multiple independent runs, nor does it report classification false refusal rates for non-English languages under real traffic.

• Research positioning: This is a system study on protective classifiers for frontier large language models addressing dual-use risks (especially in high-risk CBRN domains), proposing a defense that balances strong robustness with engineering deployment feasibility.
• Problem addressed: Relying solely on harmlessness alignment fine-tuning cannot resist general jailbreaks that systematically extract detailed operational instructions, and non-experts using general jailbreaks may cross technical thresholds to complete complex high-risk processes; however, existing defense solutions often struggle to be deployed in real products due to high inference latency or excessive false refusal rates.
• Method highlights: A dual protection system of Constitutional Classifiers is constructed, where natural language constitutions guide the synthesis of positive and negative samples and data augmentation, training an input classifier to intercept hidden intents and encoded communications, and training a streaming output classifier with an attached Value Head to judge token-by-token during generation and immediately truncate harmful content.
• Core experimental results:
  1. In cumulative over 3,000 hours (mean estimate 4,720 hours) of human red team crowdsourced testing, no report was able to extract harmful information reaching half the detail of an unprotected model across all 10 target queries;
  2. On a 5,681-query chemical weapons composite held-out jailbreak set, the classifier reduced the jailbreak success rate of Claude 3.5 Sonnet (New) from 86% without the classifier to below 5% (interception rate over 95%);
  3. In a 5K conversation sample of real production traffic on Claude.ai, the absolute increase in refusal rate from constitutional classifiers was only 0.38%, and relative inference overhead increased by only 23.7%;
  4. In 10K sampled jailbreak attack tests, the output classifier could intervene early in token generation, with the proportion of harmful streams not truncated in time being only 1.17%.
• Author conclusions and implications: The authors believe that defending against general jailbreaks while maintaining practical deployment is engineeringly feasible; as adversarial techniques continue to evolve, long-term safety cannot rely on a single protection alone, and constitutional classifiers must be deeply integrated with multi-layer defense mechanisms such as rapid vulnerability response and anomaly monitoring.