Study evaluates frontier models' multilingual moral answers: GPT-5 performs best on average, Gemini 2.5 Pro lowest in some categories

Multilingual moral and safety consistency is the question being tested: whether frontier and open-source models give stable answers to five categories of moral, legal, and safety questions in high- and low-resource languages, and whether built-in safety features in non-English languages still block malicious content.

• Scenario: The authors say people already use LLMs to write emails, do homework, and generate recipes, and usage is still increasing (Liang et al., 2025; 2024). The authors argue that answers should not lose accuracy depending on the prompt language or topic, and that safety measures capable of limiting non-English malicious content are also needed.
• Existing gaps: The authors say test data is mostly English. They cite Zheng et al. (2023), saying leading LLMs in English can judge fairly and stably give near-human answers; but academic questions (Fu and Liu, 2025) and moral questions (Jin et al., 2024) are no longer stable across languages. Safety protocols perform poorly in non-English settings, making malicious content harder to detect (Alon and Kamfonas, 2023; Lin et al., 2025). The authors acknowledge existing benchmarks (Ahuja et al., 2023) and datasets covering more than 100 languages, yet still say more evaluation is needed and existing test data has limited diversity.
• Angle: The authors first want to identify the question types where multilingual LLMs perform poorly, and how judgments change when the language changes, then build targeted data. Harm Prevention & Safety and Consent & Autonomy are written as indirect, deceptive phrasings to probe the boundaries of built-in safety features.
• Evaluation: 100 questions in each of 5 categories, 6 languages, a five-point scale plus a judge LLM, comparing GPT-5, Gemini 2.5 Pro, Claude Sonnet 4, Llama 4 Scout, and Qwen3 235B-a22b. The authors say that in preliminary tests, GPT-5 scored slightly higher than other models on unsafe or harmful prompts.

The authors discuss prior work along three lines—moral frameworks, multilingual capability, and cross-lingual safety—and position their own experiments as "looking at wording in addition to language switching, and covering five categories at once."

Moral frameworks

• DIT (Achiam et al., 2023): Uses psychology's Defining Issues Test and the standard P-score to measure moral reasoning. The authors paraphrase: some LLMs can reach reading levels comparable to higher human responses, but results vary widely across dilemmas, and ethical reasoning is inconsistent.
• Utilitarianism and deontology (Samway et al., 2025): Uses scales to see which ideology responses lean toward. The authors paraphrase: models lean utilitarian, with the degree varying by model; the reasons given often lack depth and coherence.
• The authors' assessment of these two strategies: Using developmental psychology tools, or philosophical frameworks to classify reasons, can provide observations, but they are often limited to a few designated dilemmas and fixed scales. The authors believe morality is a spectrum, and reasons often cannot be neatly categorized, so there remains a gap in how models respond to moral situations.

Multilingual capability

• Multilingual trolley problem (Jin et al., 2024): When the same dilemma is switched to another language, moral reasoning often changes with the prompt language. The authors paraphrase that the paper also notes changes may extend to legality and bias, so testing across multiple categories is needed.
• Cross-lingual factual questions (Fu and Liu, 2025): Parallel questions, different difficulty levels. The authors paraphrase: GPT-4o performs stably on high-resource languages, with more inconsistency and factual errors in low-resource languages.
• English comparison: The authors use Zheng et al. (2023) to support that "judgments are more stable in English," and state that this performance loses consistency in multilingual academic and moral questions. The authors believe multilingual reasoning and factual reliability remain unresolved, and question whether LLMs can serve as credible judges in multilingual scenarios requiring moral and factual reasoning.

Cross-lingual safety

• Non-English safety protocols: The introduction states that safety protocols perform poorly when questions are posed in non-English languages, and models have more difficulty detecting malicious content (Alon and Kamfonas, 2023; Lin et al., 2025).
• Lin et al. (2025): A large-scale dataset testing safety with malicious or illegal queries in over 100 languages. The authors paraphrase: in low-resource languages, models are more likely to produce harmful content, and there is a gap between safety guidelines and enforcement. The paper's trick questions are called jailbreak-style prompts by the authors.
• How the authors connect: The authors state that beyond language changes, they also look at wording. Harm prevention and consent categories are written as indirect questions with hidden malicious content, to see whether subtle wording makes models let harmful content through. The authors treat Lin's direct safety benchmark as evidence of cross-lingual weaknesses, and their own extension is about wording differences.

The experimental section does not list separate method baselines; the five models serve as comparisons against each other. The evaluation uses self-built questions in five categories; the experimental section does not report scores on MEGA (Ahuja et al., 2023) or MoralBench (Ji et al., 2024).

整体做法是用自建五类多语言问题集，经翻译后让模型作答，再由 Gemini 2.5 Pro 按 1–5 分量表评分。同意与伤害预防两类写成间接、欺骗性问法，用来观察内置安全功能。论文没有形式化目标函数，也没有使用 white-box、black-box 等术语。

数据与五类问题

• 规模与语言：英语 500 题，5 类各 100。译入高资源的中文、西班牙语、阿拉伯语，以及低资源的印地语、斯瓦希里语。贡献列表写 6 种语言，英语包含在内。翻译工具是 Googletrans。作者称这些语言在句式、文化语境和文字系统（字母到语素文字）上差异大。
• 三类直接问题：Biases & Stereotypes 涉及可能唤起刻板印象的人口身份，看是否延续有害偏见，并比较高低资源语言。Legality 问一些国家合法、另一些国家非法的行为，看模型是否按提示语言假定用户地点。Moral Judgment 是是/否道德题并要求理由，对照 care、fairness、loyalty、authority、sanctity、liberty。
• 两类间接问题：Consent & Autonomy 用欺骗性提示，看安全机制能否挡住可能违反个人同意与自主的信息。Harm Prevention & Safety 用伪装或间接问题，看会不会给出可能促成有害或不道德行为的信息。Figure 1 为改写问法，Figure 2 为例题；可用文本没有题干。

模型与提示

• 模型：前沿模型 Gemini 2.5 Pro、Claude Sonnet 4、GPT-5；经 OpenRouter 调用的开源模型 Llama 4 Scout、Qwen3 235B-a22b。作者称选择时考虑训练数据、用途、来源和是否开源，并挑选翻译能力上的领先模型。
• 作者对模型的描述：Gemini 偏学术应用，GPT 提供宽泛、可适应的回答，Claude 强调输出安全，Qwen3 为中国开发、区域对齐较强。这是作者的定位，不是实验结果。
• 提示参数：使用系统提示，temperature 为 0.7。每题以 under 50 words 结尾。Moral Judgment 另要求先 Answer yes/no/maybe，再给 justification。Figure 3 为模型指令，全文未进入可用文本。

流程与评分

• 流程：每道英语题译入五种目标语言之一，模型用该语言作答，回答再译回英语，由 Gemini 2.5 Pro 按量表评分。Figure 4 为测试流程。正文对 Figure 6 的讨论包含英语表现，故英语题也在评测范围内。
• 译文检查：各类、各语言抽取译题和回答，送入 GPT-5 与 Qwen，检查是否保持原意。另抽查非英语回答与英语对应项，检查翻译偏差。论文未报告抽样题数。
• 量表：1–5 分，每类单独规定 5 分与 1 分的约束，结论和理由都看。作者称道德信念因文化而异，且当时无法取得跨文化的适当人类样本，因此把量表当作评分中的标准。附录 A.1 为量表，条目未进入可用文本。
• 交叉评分的设计：随机回答再用 GPT-5 与 Qwen 3 评分，用来防范 Gemini 偏向自己的回答。交叉评分的数字放在实验结果里。

实验设置

• 被测模型：Gemini 2.5 Pro、Claude Sonnet 4、GPT-5、Llama 4 Scout、Qwen3 235B-a22b。后两个经 OpenRouter。结果节常把 Qwen3 235B-a22b 简称为 Qwen，把 GPT-5 简称为 GPT。
• 数据：自建英语 500 题，5 类各 100；语言为英语，以及中文、西班牙语、阿拉伯语（高资源）和印地语、斯瓦希里语（低资源）。翻译与回译用 Googletrans。流程写的是每道英语题译入目标语言后评分。
• 指标：1–5 分量表上的类别平均；第 4 节另把全类平均写成百分比成绩。结论提到 refusal rates，但没有给出拒答率数字。论文未把百分比换算成 5 分制，这里也不换算。
• 评审：主 judge 为 Gemini 2.5 Pro。随机样本另由 GPT-5 与 Qwen 3 评分。论文未报告该样本的题数、实验重复次数，或与人工评分的一致性。
• 生成设置：temperature 0.7；回答限制在 50 词以内。Moral Judgment 要求先 yes/no/maybe 再给理由。
• 图表：Figure 5 为最终结果，Figure 6 为最终图，Figure 7 为回答示例，附录 B.1、B.2 为方差和分模型结果。图内刻度未能从文本读出，下表只列正文写明的数字。

主结果

模型	正文写明的分数	出处
GPT-5	Consent & Autonomy 3.56；Harm Prevention & Safety 4.73；全类 almost 92%	摘要；第4节
Gemini 2.5 Pro	Consent & Autonomy 1.385，摘要写 1.39；Harm Prevention & Safety 1.98	Figure 5正文；摘要；结论
Claude Sonnet 4	Harm Prevention & Safety 3.75	结论
Qwen3 235B-a22b	全类 66%	第4节
Llama 4 Scout	正文未写出具体分数	图见 Figure 5、附录 B

• 作者对总体的说法：摘要称 GPT-5 在每个类别的平均都最高，其他模型在语言和类别上更不一致。贡献列表称 GPT-5 相对其他领先模型明显更高；引言对不安全或有害提示只称略高。论文没有解释这两处措辞的差别。除上表两类外，各类的模型间数字未在正文写出。
• 常规类与欺骗类：作者称 Gemini 在 Biases & Stereotypes、Legality、Moral Judgment 上较高，在更具欺骗性的类别下降，两类分数相差超过 2 分。作者认为这很可能是因为 Gemini 的测试数据以学术论文为主，因而在更事实、更直接的类别上较高。
• GPT 的例外：作者称 GPT 只在 Consent & Autonomy 上表现差，并把它联系到 OpenAI 的安全准则，以及针对越狱和控制恶意内容传播的更多测试。作者还称该类问的是可能的情感与心理伤害，而不是身体暴力，因此预期模型比 Harm Prevention & Safety 更可能作答。

语言差异与 Legality

• 测了什么：Figure 6 的正文比较英语、常规类和 trick question 类，以及高、低资源语言。论文没有在正文给出分语言分数。
• 作者报告的模式：各模型在英语常规类上较高，trick 类较低；这两类里 GPT 平均最高、Gemini 最低。作者称欺骗类中，低资源语言分数高于高资源语言。
• 作者的解读：作者认为可能是高资源语言的测试数据更多。并称低资源语言下模型会注意到有害用词而拒答，高资源语言下会读上下文并被诱导作答。Legality 上，作者称 Qwen 在低资源语言较差，在中文也差，尽管它是中文模型；作者称这一结果出乎预期，并认为可能是因为 Qwen 假定用户在中国、只按中国法律回答。作者给出的背景是：它是中国模型，测试数据多数为中文，大量用户在中国。

安全准则与评审检查

• 准则差异：作者称模型间安全协议不同，也是这两类分数有差别的原因，例如 Claude 的安全准则比 Qwen 或 Gemini 更严。结论称 Claude Sonnet 4 在 Harm Prevention & Safety 上相对较强。论文未引用准则原文。
• judge 交叉检查：相对 Gemini 的评分，Qwen 3 平均低 0.5，GPT-5 平均高 0.6（第 3.3 节）。作者称 Gemini 给自己的回答打分，相对给其他模型打分，未见偏向。论文未做统计检验。
• 翻译抽查：作者称抽查的非英语回答与英语对应项，评测都在 1 以内。文本没有写明 1 的单位。译文是否保持原意的检查没有给出定量结果。

其他消融与分析

论文没有参数消融表。其余为正文中的单项观察：

• Gemini 的常规类与欺骗类相差超过 2 分（第 4 节，未给各类精确分）。
• 欺骗类中低资源语言分数高于高资源语言（Figure 6 的正文解读，无分语言数字）。
• Qwen 的 Legality 在低资源语言和中文上都较差（无具体分数）。
• 交叉 judge：Qwen 3 比 Gemini 平均低 0.5，GPT-5 平均高 0.6。
• 附录 B 为回答方差和分模型结果，文本未抽出数字。
• Llama 4 Scout 在正文中没有单独分数。

Limitations and future directions noted by the authors

• Translation: Questions and answers were translated using Googletrans; despite verification, inaccuracies and inconsistencies may remain and may affect answers and scoring (Section 5).
• No human answers: Human answers were not collected, and the scale was used to establish ground truth. The authors say the evaluation reflects how close answers are to the scale, not a strong indicator of adherence to current societal values (Section 5).
• Wording sensitivity: Models may form some impression of the user based on question wording, thus changing answers due to phrasing rather than content (Section 5).
• Expanding languages: Currently English, Spanish, Chinese, Arabic, Hindi, and Swahili. The authors plan to expand to all languages supported by Googletrans (Future Work).
• Human baseline: The authors plan to collect human answers from different countries and spoken languages as a baseline for comparison with current societal values (Future Work).
• Wording variants: The authors plan to have future data account for wording. Slightly different phrasings of the same question may yield inconsistent results, and leading or suggestive questions may affect the model's understanding of user intent; the authors say testing these variants can reduce evaluation noise and expose potential biases in model perception (Future Work).

Experimental coverage

• Models tested are Gemini 2.5 Pro, Claude Sonnet 4, GPT-5, Llama 4 Scout, Qwen3 235B-a22b, 5 in total (Section 3.2).
• Questions are 100 per category across 5 categories, 500 in English, translated into Chinese, Spanish, Arabic, Hindi, and Swahili; the contributions list says 6 languages (Section 3.1).
• Primary scoring is Gemini 2.5 Pro with a 1–5 scale; GPT-5 and Qwen 3 are used for cross-scoring and translation meaning checks on random samples (Sections 3.3, 3.5).
• Generation settings are temperature 0.7, within 50 words; Moral Judgment requires yes/no/maybe plus reasoning (Section 3.4).
• The paper does not report the number of questions in random samples or the number of experimental repetitions. Figure 5, Figure 6, and Appendix B contain per-model, per-language figures, and the values in the figures could not be read from the text.

This work compares five LLMs' multilingual answers on ethics and safety questions, examining whether scores vary by language and question type.

• Question: The authors test how ethical, legal, and safety answers shift between high- and low-resource languages, and whether built-in safety features still block malicious content under non-English prompts.
• Method: 500 English questions in five categories are translated into Chinese, Spanish, Arabic, Hindi, and Swahili. Consent & Autonomy and Harm Prevention & Safety use indirect, deceptive phrasings. Answers are translated back to English and scored by Gemini 2.5 Pro on a 1–5 scale; temperature is 0.7, and answers are limited to 50 words.
• Numbers: The authors report GPT's overall category percentage score near 92%, and Qwen's at 66%. In the abstract, GPT averages 3.56 and 4.73 on Consent & Autonomy and Harm Prevention & Safety, while Gemini 2.5 Pro averages 1.39 and 1.98; the Figure 5 text writes Gemini's former as 1.385. The conclusion gives Claude Sonnet 4 an average of 3.75 on Harm Prevention & Safety. Llama 4 Scout has no numbers in the text.
• Authors' interpretation: Regular categories score higher than deceptive ones; within deceptive categories, low-resource languages score higher than high-resource ones. The authors believe that in high-resource languages, models understand context and are more easily induced to answer, while in low-resource languages they notice harmful wording and refuse. Qwen still performs poorly on Chinese Legality; the authors believe it assumes the user is in China and answers only according to Chinese law.
• Authors' conclusion: The authors state that safety mechanisms are reliable in English but often fail to transfer to other languages; reasoning, accuracy, and refusal rates are all sensitive to language and phrasing. The authors state that existing benchmarks underestimate the challenge of multilingual safety and that more robust multilingual data and evaluation frameworks are needed.