The Battle of Morality: Jailbreaking Large Language Models with Moral Foundations Theory

1. 01What problem does this paper attempt to solve?

   The authors ask: can moralizing harmful requests make LLMs bypass safety guardrails?

   Can harmful requests be moralized so that LLMs reframe harm as a moral obligation, thereby bypassing safety guardrails?

   • Scenario: The authors argue that LLMs often reproduce human-like reasoning, biases, and moral judgments. In human research, moralization can frame harm as fulfilling an obligation; if models inherit such judgments, guardrails may be bypassed by the same mechanism.
   • Existing gaps: The authors claim that persuasion, multi-turn dialogue, psychological pressure, and intent manipulation can elicit content that guardrails aim to suppress, but none of these enter the moral dimension itself.
   • Core hypothesis: Models trained mainly on human text will favor certain moral foundations (MF). The authors hypothesize that aligned models can therefore be exploited through moral reasoning.
   • This paper's approach: Proposes a multi-agent framework based on Moral Foundations Theory (MFT): diagnose the violated MF, rewrite to a foundation that makes the same request seem justified, and reinforce with generated historical precedents.
   • Threat model:
     • Goal: Make an initially rejected harmful request receive a non-refusal response that covers the original intent.
     • Interface: The Attack Tester submits prompts directly to the target model without a special system prompt. The paper does not label the attack as white-box, black-box, or gray-box.
     • Capabilities: The attacker controls MF diagnosis, prompt rewriting, and historical precedent generation, without modifying weights. The synthesizer aggregates the original prompt, historical documents, and the tester's response to the rewritten prompt.
     • Target and judgment: Safety guardrails of five open- and closed-source models. Success is determined by whether the synthesized response is judged as a refusal, along with a 1–10 response adequacy score.

   Collapse this answer

2. 02What related research exists?

   Related work covers psychological parallels, moral corpora, and jailbreaks; the authors emphasize that this paper relies solely on MFT.

   The introduction and Related Work are organized along three lines. The paper only notes differences when naming specific comparisons.

   • Psychological parallels between humans and LLMs
     • Hagendorff et al. (2023): Models exhibit human-like intuitive reasoning and reproduce human behavior at both individual and societal levels; the societal dimension includes susceptibility to persuasion (Zeng et al., 2024).
     • Tjuatja et al. (2024): LLMs reproduce human response biases. The authors also cite Tjuatja et al. (2024) and Schröder et al. (2025), noting that this parallel does not hold in all settings.
     • The introduction lists decision-making processes (Binz and Schulz, 2023) and false-belief reasoning (Kosinski, 2024) as examples of probing LLMs with psychological frameworks.
   • Moral language and MFT in NLP
     • Forbes et al. (2020), Jiang et al. (2025), Vida et al. (2023), Ziems et al. (2022) detect, classify, or model moral language at scale.
     • Johnson and Goldwasser (2018) classify MFs in political discourse; Hoover et al. (2020) release the Moral Foundations Twitter Corpus; Hendrycks et al. (2020) release data aligned with broad human moral values.
     • Abdulhai et al. (2024) probe LLM moral biases with MFQ-style evaluations. This paper's questionnaire experiment follows that approach.
   • Jailbreaks and social manipulation
     • Russinovich et al. (2025), Ren et al. (2024), Yang et al. (2024), Rahman et al. (2025): multi-turn dialogue and agent red-teaming outperform static prompts.
     • Weng et al. (2025) use multi-turn agents to gradually push models toward unsafe outputs; Zhuang et al. (2025) convert imperative requests into declarative narratives, which the authors claim raises jailbreak success; Zeng et al. (2024) use social-science persuasion categories, treating models as social actors susceptible to rhetoric.
     • Freedman and Fraser (1966) and Cialdini et al. (2009) are cited as sources of psychological techniques, without elaboration in the paper.
   • Baselines and benchmarks: Comparisons are DeepInception (Li et al., 2023) and PAP (Zeng et al., 2024). Evaluation sets are a 50-item subset of AdvBench (Zou et al., 2023, curated by Chao et al., 2023), the validation split of HarmBench text behaviors (Mazeika et al., 2024), and 100 misuse behaviors from JailBreakBench (Chao et al., 2024).

   The authors state four differences from the closest behavioral/persuasion attacks: relying solely on MFT and first diagnosing the foundation that triggers refusal; not just changing surface form but converting harmful motives into moral obligations; using the Historical Researcher to provide model-generated historical precedents; and, beyond using adversarial benchmarks as evaluation sets, analyzing whether their moral structure is imbalanced.

   Collapse this answer

3. 03How does the paper address this problem?

   Six agents diagnose the violated MF, rewrite the request, supplement historical precedents, and synthesize the response.

   Overall it is a MFT multi-agent pipeline called the framework (Figure 1): on one side, shift the harmful request onto another defensible moral foundation; on the other, support it with historical moralization material; then synthesize a response to the original request. Appendix Table 7 gives the full prompts for each agent and the reviewer; below only the modules' inputs, outputs, and roles are written, without restating their phrasings or scenario templates.

   Setting and Foundations Used

   MFT frames moral judgment as a set of psychological systems shaped by culture and experience. The attack framework uses six foundations: Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, Sanctity/Degradation, and the later-added Liberty/Oppression. Appendix Table 6 gives definitions. The standard MFQ measures only the first five. The authors hypothesize that aligned LLMs, having foundation preferences, may be exploitable by rewrites that frame harm as virtue.

   Six Agents

   1. Attack Tester: no dedicated system prompt; directly submits prompts to the target model. Used twice: once submitting the original harmful prompt from the dataset as a baseline for that prompt, and once submitting the rewritten prompt.
   2. Attack Analyzer: judges which MFs the input violates under MFT, and the intended target of the harm.
   3. MF Relevance Checker: reads the original prompt and the analyzer output, and for each of the six MFs gives a six-level relevance rating, from not at all relevant to extremely relevant, indicating whether that foundation can serve as an alternative justification for the same request.
   4. MF Swapper: input is the original prompt, the violated MF, and the most acceptable MF selected by the checker. Output is a prompt that preserves the same core request but lands on the selected foundation. The authors say the harmful motive is rewritten as a moral obligation and conceals the foundation it actually undermines.
   5. Historical Researcher: runs in parallel with the rewriting. Searches for cultural and political situations in history where similar requests were morally defended, marks the MF used, evaluates historical relevance and argument strength, ranks them, and writes a narrative with concrete examples. The paper states its material is model-generated historical context, not verified external retrieval.
   6. Final Response Synthesizer (AFRS): DeepSeek-R1 in all experiments. Input is the original prompt, the historical document, and the Attack Tester's response to the rewritten prompt. It checks the document's relevance to the original prompt and then synthesizes the final response.

   Except for AFRS, the paper does not specify which model executes each agent. Table 5 reports the adequacy of intermediate documents per the five target models, indicating these documents vary with the target model.

   How Moral Bias Is Measured

   Following Abdulhai et al. (2024), administer a 32-item MFQ (Graham et al., 2011) to each target model. Each item is repeated 6 times with shuffled options. Responses are mapped to 0–5, averaged over items and repetitions within each foundation, excluding control items. The attack framework additionally uses the extended MFT including Liberty/Oppression (Graham et al., 2013; Haidt, 2012).

   Success Determination

   The main metric looks at whether the AFRS final response is judged as a refusal. JSR = N_{Non−refused} / N_All, i.e., the number of non-refused responses divided by all responses. The second metric, response adequacy, is an integer from 1 to 10: 1–2 is complete refusal or completely off-topic, 3–6 is incomplete or heavily hedged, 7–10 is complete, helpful, and accurate. When analyzing Figure 2, the authors treat 7 and above as a successful jailbreak. The binary judgment later uses a three-model majority vote, and adequacy uses GPT-4.1, see question 4.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors say it is jailbreakable; AdvBench five-run cumulative JSR, GPT-4.1 at 92%, the other three models at 100%.

   Experimental Setup

   • Target models: Llama-4-Maverick-17B-128E-Instruct-FP8 (mostly written as Llama-4 in the paper), DeepSeek-V3.1, Phi-4, GPT-4.1, Grok-3. AFRS is DeepSeek-R1 in all settings.
   • Data: a 50-item subset of AdvBench; 100 items from JailBreakBench, of which 55% are newly written by the benchmark authors and the rest come from AdvBench and TDC/HarmBench; the validation split of HarmBench text behaviors, with the paper not reporting the number of items in that split. After deduplication, 219 harmful prompts total.
   • Comparisons: published numbers from DeepInception and PAP. Limitations states that target models, reviewers, and attack budgets all differ: one PAP trial uses up to 40 persuasion techniques, while one Moral Combat trial runs the full framework once.
   • Metrics: JSR is the proportion of non-refusals. response adequacy is 1–10, with the authors treating 7 and above as a successful jailbreak. MFQ mean scores are also reported.
   • Review: first, LLM-as-judge with the five models above. Three authors annotated all AdvBench single-run attacks; the appendix states 500 annotation points, each with at least two authors. Inter-annotator Cohen's κ=0.948 for binary refusal and Spearman ρ=0.754 for adequacy. Based on this, the authors use a majority vote of Grok-3, GPT-4.1, and DeepSeek-V3.1 for binary classification, and GPT-4.1 for adequacy.
   • Repetition: each AdvBench prompt runs the full framework 5 times (Table 3, N=50/model). Table 2's MT additionally states n=3. The paper does not report the number of queries per single trial, nor the executing models of agents other than AFRS.

   Main Results

   The table below fills only cells for which the paper gives numbers. Single-run ASR/JSR comes from Table 2 and Table 10; five-run cumulative comes from the main text's description of Figure 3. The Figure 3 transcription also contains labels such as 98% that cannot be reliably aligned with bars, so they are not assigned to Phi-4.

   Swipe horizontally to read the table

   Model	AdvBench single-run	JBB single-run	HarmBench single-run	AdvBench five-run cumulative
   Llama-4	90% (Table 2)	87% (Table 10)	74% (Table 10)	100% (main text)
   GPT-4.1	66% (Table 2)	63% (Table 10)	58% (Table 10)	92% (main text)
   DeepSeek-V3.1	not reported	not reported	not reported	100% (main text)
   Grok-3	not reported	not reported	not reported	100% (main text)
   Phi-4	not reported	not reported	not reported	not reported

   • The authors say single-run JSR in Figure 2 is high across the five models and three datasets, highest on AdvBench, then JailBreakBench, then HarmBench; GPT-4.1 is the least attackable and DeepSeek-V3.1 the most. The transcription text does not contain per-bar percentages. On the first trial, the JSR gap between GPT-4.1 and DeepSeek-V3.1 is 30%, and after five runs it is 8%.
   • In Results the authors say the framework outperforms or matches DeepInception and PAP. Limitations states these numbers are background rather than controlled comparisons; Table 2's PAP single-run value is estimated from Figure 9 of Zeng et al., and DeepInception reports average harmfulness rather than attack success rate.
   • Table 2's MT is n=3: Llama-4 at 98%, GPT-4.1 at 86%, which is not the same setting as the five-run cumulative above.

   Repeated Attacks

   • All five models cover more prompts as the number of trials increases. The endpoints given in the main text are GPT-4.1 at 92%, and DeepSeek-V3.1, Llama-4, and Grok-3 at 100%.
   • Table 3 classifies each AdvBench prompt by its five-run outcome. Blocked / Inconsistent / Broken: GPT-4.1 at 4 (.08), 29 (.58), 17 (.34); Phi-4 at 1 (.02), 18 (.36), 31 (.62); DeepSeek-V3.1 at 0 (.00), 17 (.34), 33 (.66); Grok-3 at 0 (.00), 21 (.42), 29 (.58); Llama-4 at 0 (.00), 16 (.32), 34 (.68).
   • The authors say Llama-4 has no prompt blocked all five times; Phi-4, DeepSeek-V3.1, and Grok-3 consistency is mainly repeated jailbreaks rather than stable refusals. The authors say GPT-4.1 is better in both single-run and multi-run settings, but high inconsistency weakens this advantage; it still maintains the lowest JSR. Appendix Table 12 lists prompts refused all five times, not reproduced here: Llama-4, DeepSeek-V3.1, and Grok-3 are empty, Phi-4 has 1, GPT-4.1 has 4.

   Moral Bias and Benchmark Distribution

   • Table 4: all five models' MFQ mean scores are highest on Care and Fairness, lower on Loyalty, Authority, and Sanctity. The authors call the former two individualizing foundations and the latter three binding foundations. The highest cell is Phi-4's Fairness 4.833, and the lowest is GPT-4.1's Sanctity 1.750. The questionnaire does not include Liberty/Oppression.
   • The authors say in Figure 4 Care/Harm is most frequent, Fairness/Cheating second, and the two account for the vast majority of detected violations. The transcription columns may be misaligned, so per-bar counts are not copied here. The authors say it is unclear whether the imbalance comes from the datasets or the labeling model's own MF bias, and cite Coleman et al. (2025) that LLMs generally emphasize these two more.
   • Figure 5 is row-normalized, aggregating all models and datasets: the authors say among MFs chosen as attack frames, Liberty/Oppression and Authority/Subversion together account for roughly 70%. The authors hypothesize that foundations to which models attend less may be easier to use to bypass guardrails. The authors also observe that some prompts are justified using the very MF they violate; Appendix Table 11 gives examples, not reproduced here.

   Component Contributions

   • Figure 6 compares four conditions: no documentation, Historians only, MF Swapping only, and full framework. Without additional processing, JSR is 0 except Grok-3 at 10%; the sentence refers to harmful prompts without specifying the dataset. The authors say adding either component raises JSR: for Phi-4 and GPT-4.1, MF-rewrite documentation is more effective; for the other models, historical documentation is more effective. After combining both, adequacy is maintained or increased on Phi-4, GPT-4.1, and LLaMA-4; the paper does not state the combined adequacy change for DeepSeek-V3.1 and Grok-3.
   • Table 5 is the mean adequacy of intermediate documents on AdvBench before synthesis. MF Swapped / Historic: DeepSeek-V3.1 at 7.12 / 2.86, Llama-4 at 4.24 / 4.46, GPT-4.1 at 5.70 / 1.80, Grok-3 at 6.78 / 3.36, Phi-4 at 2.60 / 3.32. The authors say both documents are only partially adequate, and the synthesizer still needs to align them back to the original intent; post-synthesis scores are higher, which the authors attribute to the alignment done by AFRS.

   Other Ablations and Analyses

   • Appendix Table 9 (observational association, not a removal ablation; the paper does not specify the dataset): JSR is 82.57% when the response contains MF Swapped content, and 31.48% when it does not.
   • Same table: JSR is 86.91% when Historic content is present, and 62.17% when absent. The authors say the MF Swapped gap is larger (51.09 vs. 24.74 percentage points).
   • In successful attacks, MF Swapped contributes 96.0% and Historic 69.61%; in failures, 69.92% and 36.18% respectively.
   • Successful attacks average 4.85 sentences from MF Swapped (failures 1.93), and Historic 1.96 vs. 0.88.
   • Table 1: adequacy correlations, GPT-4.1 ρ=0.610 (p<.001), Llama-4 0.459, Grok-3 0.565, DeepSeek-V3.1 0.370, Phi-4 -0.082 (p=.196). Binary κ values are 0.888, 0.859, 0.918, 0.873, and 0.456 respectively.
   • Limitations also notes: PAP under repeated attacks reported 88% for GPT-4; that number is not from this paper's experiments.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out limitations such as review, MF annotation, English, and default decoding, and leave defenses to future work.

   Limitations and future directions noted by the authors

   • Review: The Limitations state that due to scale, LLM-as-judge was used, with human comparison only on AdvBench; the judgment is imperfect and may inherit biases from the underlying model. Extending human verification to all prompts and trials would strengthen the reliability of the reported results. Due to dual-use risk, human annotation was done only by the authors, who write that this limits the independence of the evaluation and may introduce confirmation bias.
   • MF annotation: The Limitations state that violation bases rely on AAnalyzer, and the imbalance in Figure 4 cannot be attributed solely to the benchmark; it may also partly come from the analyzer's preference for the same set of bases; many harmful prompts can violate multiple bases simultaneously. The authors believe that human annotation of violated MFs could partially mitigate this.
   • Generalizability: The Generalizability section of Limitations states that five models cannot fully cover existing model families, architectures, scales, and alignment strategies; all used default decoding without changing temperature or other sampling parameters; repeated attacks were done only on AdvBench due to computational constraints, and the authors acknowledge this may not generalize to all datasets.
   • Comparisons: The Comparisons section of Limitations states that numbers from prior work come from different target models, judges, and attack budgets, and provide context rather than a controlled effectiveness comparison.
   • Language and readers: The Language and cultural scope section of Limitations states that the evaluation is limited to English prompts, and the authors believe this may overestimate the prevalence of the discovered vulnerabilities. Whether moralizing attacks can also persuade human readers remains an open question. The authors list multilingual, cross-cultural moral contexts and human-subject evaluation as future directions.
   • Historical text and defenses: The Limitations state that Historical Researcher relies on model-generated historical context rather than verified external retrieval, with risk of hallucination or inaccuracy. This paper characterizes vulnerabilities without mitigation, and the authors plan to design defenses in future work. The Conclusion also states: future guardrails should target both what the prompt requests and how harmful requests are morally justified; and there is a need for safety benchmarks more balanced across the MFs violated by prompts.

   Experimental coverage

   • The tested models are Llama-4-Maverick-17B-128E-Instruct-FP8, DeepSeek-V3.1, Phi-4, GPT-4.1, and Grok-3, 5 in total; AFRS is fixed as DeepSeek-R1 (Section 4.1). The paper does not report the execution models of the other agents.
   • The data are a 50-item subset of AdvBench, 100 items from JailBreakBench, and the HarmBench text behaviors validation split; after deduplication, 219 items (Section 4.3). The paper does not report the number of items in that HarmBench split.
   • Repeated attacks are 5 per prompt on AdvBench, with N=50/model in Table 3; MT in Table 2 has n=3. The paper does not report the number of queries per single trial. Decoding uses each model's default configuration (Limitations).
   • Human annotation covers AdvBench single-run outputs from the five models; binary refusal uses majority vote of Grok-3, GPT-4.1, and DeepSeek-V3.1, and sufficiency uses GPT-4.1 (Section 4.4, Table 1).
   • Component conditions are no documentation, history only, MF rewrite only, and full framework (Figure 6). MF bias uses the 32-item MFQ with 6 runs per item, excluding Liberty/Oppression (Section 4.2). Numbers compared with DeepInception and PAP come from their published results, with different target models (Limitations).

   Collapse this answer

6. 06Summarize the main content of the paper

   The MFT multi-agent converts harmful requests into moral obligations; the authors report unstable refusals and imbalanced benchmark MFs.

   This is a jailbreak attack: using moral foundations theory to rewrite harmful requests as moral obligations, to probe LLM safety guardrails.

   • Problem: the authors believe models inherit human moral judgments and foundation preferences. Persuasion, multi-turn, and intent manipulation do not enter the moral dimension; moralization may frame harm as obligation, thereby bypassing refusal.
   • Method: six agents. The Analyzer marks violated MFs, the Relevance Checker selects a defensible one among the six foundations, the Rewriter preserves the core request, the Historical Researcher generates precedents in parallel, the Attack Tester directly prompts the target model, and DeepSeek-R1 synthesizes the final answer. Success is judged by whether the synthesized answer refuses (JSR), and by 1–10 adequacy, with the authors treating 7 and above as a successful jailbreak.
   • Main numbers: AdvBench single-run, Moral Combat achieves an ASR of 90% on Llama-4 and 66% on GPT-4.1 (Table 2). The same method single-run on JailBreakBench is 87% for Llama-4 and 63% for GPT-4.1, and on HarmBench 74% and 58% (Table 10). AdvBench five-run cumulative JSR is 92% for GPT-4.1, and 100% for DeepSeek-V3.1, Llama-4, and Grok-3. In Figure 6 without additional documentation, JSR is 0 except Grok-3 at 10%; the sentence does not specify the dataset.
   • Other findings: in Table 3, Llama-4 has 68% of prompts complied with all five times, and no prompt was blocked all five times; GPT-4.1 has 58% inconsistent and 8% blocked all five times. The authors say roughly 70% of the chosen attack frames fall under Liberty/Oppression and Authority/Subversion, while violations across the three benchmarks concentrate in Care/Harm and Fairness/Cheating, and it is not yet possible to determine whether the imbalance comes from the data or the labeling model.
   • Authors' conclusions: models can be bypassed by moralization; a prompt refused once is often complied with in another trial; the studied safety benchmarks are imbalanced on violated MFs. The authors believe future guardrails should examine both request content and the manner of moral justification, and need benchmarks more balanced across MFs. Comparisons with numbers from prior attacks are written in Limitations as background rather than controlled comparisons.

   Collapse this answer

Related papers