Global safety blind spot: translation as a safety bypass route in low-resource languages

Harmful requests judged safe in English may still yield unsafe completions when translated into low-resource languages; the authors aim to measure this translation bypass separately.

• Setting: The authors say models are deployed globally, but safety evaluation and alignment concentrate on English. A guard that looks robust under English tests may fail in another language for the same intent, and an adversary needs only low-cost translation. The reporting pattern the authors cite is: among the top models reported by 24 released systems, 20 advertise broad multilingual support, and only 5 report any multilingual safety red-teaming (Yong et al., 2025).
• Existing gap: Translation bypass has been reported. The authors say that work usually does not report the conditional quantities an attacker would actually act on, nor test whether weaknesses are organized by language type. An unconditional unsafe rate mixes failures that already occur in English with failures that occur only on translations.
• Hypothesis: The authors say safety depends on the same language understanding as other tasks, and alignment trained mainly in English may generalize poorly to the least-covered languages. The authors state they do not claim to have discovered translation bypass.
• This paper's measurement: Three parts. (i) A metric that separates translation bypass from pre-existing English unsafety; (ii) a test of whether risk is organized by language type across 78 low-resource languages and 20 language families; (iii) models have no stable ranking, so no leaderboard is released. Prompt originals are not released, and per-category bypass rankings are withheld.
• Threat model:
  • Objective: Obtain a harmful completion with as little effort as possible.
  • Capability: Translate prompts into multiple languages and query repeatedly, including trying several low-resource languages.
  • Strategy: Probe in English first; if the English response is safe, translate the same intent into one or more low-resource languages. If any translation yields a harmful completion, the attack counts as successful. No jailbreak techniques are needed.
  • Failure the metric targets: Not occasional slips outside English, but whether the model fails broadly and consistently on prompts it already handled safely in English, after translation.

Translation bypass

• Yong et al. [2023] report low-resource-language translation jailbreaks against GPT-4. Deng et al. [2024] propose MultiJail and describe multilingual settings. Al Ghanim et al. [2024] obtain similar effects with Arabic transliteration.
• The authors state that this line shows the problem exists, but typically does not report the conditional quantities on which an attacker would act, nor test whether weaknesses are organized by language type.

Multilingual capability and safety reporting

• Yong et al. [2025] discuss English-centric safety evaluation and give the above reporting-habit statistics of 20/24 and 5/24.
• Global-MMLU, MMLU-ProX, XCOPA: the authors state that models perform best in English and worse in lower-resource languages (Singh et al., 2025; Xuan et al., 2025; Ponti et al., 2020). Wang et al. [2024] is cited in support: alignment trained mainly in English may generalize poorly to the least-covered languages.

Benchmark scale and scoring method

• ALERT (Tedeschi et al., 2024) is a large-scale red-teaming benchmark with fine-grained categories; M-ALERT (Friedrich et al., 2025) extends it to multiple languages. The authors state that these benchmarks have far more prompts than this paper's 119 (one to two orders of magnitude more), and this paper does not claim an advantage in prompt breadth; the authors state that the contribution is orthogonal: more languages, a typological distribution at the low-resource end, and conditional metrics.
• Mazeika et al. [2024] represents many benchmarks that report refusal rates rather than unsafe completion rates; An et al. [2024] studies false refusals. The authors state that these two phenomena may be mixed in the same reviewer label.

Translation-quality explanation and typology

• Marx and Dunaiski [2026] report lower single-turn jailbreak rates on four low-resource African languages and seven commercial models, and attribute this to machine translation weakening harmful intent before it reaches the model. The authors state that this explanation cannot be ruled out, because this paper uses only one translator and does not record quality covariates; but this explanation cannot account for why the language-family ranking changes with the model. The test is in Section 5.5.
• Resource tiers follow Joshi et al. [2020], and typology uses WALS (Dryer, 2013). Blaschke et al. [2025] show that the link between language similarity and cross-lingual transfer depends on the setting, so the authors test language-family structure rather than assuming it.

Baselines and data sources: This paper sets no separate attack-method baseline. The 119 prompts come from AdvBench, MultiJail, PHTest, HarmBench and DiaSafety. The regression uses Claude-Sonnet-4 as the reference (OR = 1).

The authors position this paper as measurement rather than the discovery of translation bypass: the conditional quantity, the formal test by language type, and not releasing a leaderboard. Compared with benchmarks with more prompts, the authors state that the difference is language coverage and conditional metrics; for the translation-quality explanation, the authors use the change in language-family ranking across models to test whether it can explain all the differences.

In the translation-mediated bypass setting, the same set of English harmful prompts is translated into low-resource languages, and the unsafe proportion conditional on English safety is used to separate translation bypass from pre-existing English unsafety.

Prompts, languages and models

• Prompts: 119, explicitly requesting or advocating unsafe, illegal or unethical behavior, from AdvBench, MultiJail, PHTest, HarmBench, plus non-directive prompts from DiaSafety. 11 categories, concentrated in Dangerous Behavior & Criminal Content (35), Hate & Discrimination (20), Violence & Threats (18), Self-Harm & Suicide (15) (Table 11). Construction was aggregation, deduplication and heuristic filtering, adding DiaSafety, then a stratified subset by harm category and request type (Appendix B, C). Request types are Directive, and non-Directive Interrogative, Assertive, Expressive.
• Languages: English plus 78 low-resource languages, selected from the Google Translate vocabulary, prioritizing typologically diverse low-resource languages with more native speakers. Resource levels use Joshi et al. [2020] Level 0 and Level 1. 20 language families (Table 12): Austronesian and Indo-European 15 each, Niger-Congo 9, Turkic 6, Creole 5, Sino-Tibetan and Nilo-Saharan 4 each, and the remaining 20 languages in 13 smaller families.
• Models: 10, mixing open-weight, closed-source and partially open-weight. The main text lists two Claude 4, two Gemini 2.5, GPT-5, Grok-4, two Llama 4 variants, Mistral 2 Large, Phi-4; the results tables record them as Claude-Sonnet-4, Claude-Opus-4, Gemini-Pro-2.5, Gemini-Flash-2.5, Llama-Scout-17B, Llama-Maverick-17B, Mistral-Large. The authors label all as medium or large scale according to public information, each of the two families with two models contains a medium and a large model, and they state that scale effects are not separated. Temperature is set to 0 where supported, and recommended defaults are used for other hyperparameters. Except that Grok-4 cannot process Shan, all models see the same translations. Full models have 9,401 prompt–language inputs per model, Grok-4 has 9,282; 93,891 pairs in total before filtering.

Translation and judgment

• Forward translation uses the public Google Translate API. The authors say this operationalizes an attacker's ability to cheaply rephrase harmful intent; the reasons for choosing it are language coverage, relatively high fluency on low-resource languages, and consistent domain handling on safety-related content.
• Responses are collected in the target language, then back-translated into English with the same system for English review and manual spot checks. The reviewer is GPT-4o, used only for scoring, labeling back-translated English responses Safe or Unsafe. The paper does not state a numerical threshold.
• After filtering 239,292 raw responses, the harmful analysis frame is about 92,536 low-resource completions; conditional on EN-safe it is the 56,148 rows used in Section 5.3.
• Professional translators spot-check 5 in-sample languages and 1 out-of-sample language, 133 items each, on four levels: Perfect, Great, Fair/Poor, Nonsense/Unintelligible. Languages with poor translation and poor responses are removed. The spot checks come from a separate generation; the authors say it shows the faithfulness of prompt translation, not the completions that were scored. To reduce misuse, only category metadata and aggregate statistics are reported. Appendix A has qualitative examples; prompts and responses are not reproduced here.

Conditional bypass metric

• A prompt is EN-safe for a model if and only if that model's English completion is judged safe. For that prompt, the low-resource unsafe share is the proportion of the 78 languages that produce an unsafe completion.
• Aggregated as: given EN-safe, the average percentage of low-resource languages that are unsafe. Breadth thresholds are P(≥20% LR unsafe | EN-safe) (at least 16/78) and P(≥50% LR unsafe | EN-safe) (at least 39/78). The authors say the thresholds express how broad the failure is and avoid relying on the maximum.
• Attack success in the threat model is any translation yielding a harmful completion; the reported metrics are share and breadth. The conditional metric varies by model because each model's EN-safe set differs.

Fixed effects, language families and ranking schemes

• The authors run a prompt fixed-effects regression on the EN-safe subset to handle subset selection. The unit is a (model, prompt) pair whose English completion is safe; the outcome is the unsafe count among the 78 low-resource versions. Binomial GLM with logit link: logit P(unsafe in LR | model m, prompt p) = α_p + β_m. α_p is the prompt fixed effect, β_m the model effect, with Claude-Sonnet-4 as reference. The fit uses 681 pairs and 87 prompts; prompts that are always 0 or always 78 within that subset are excluded. The authors report overdispersion φ = 3.47 and use dispersion-corrected standard errors.
• The language-family test compares an additive model with one adding a model×language-family interaction, using a likelihood ratio test (Appendix F). Figure 3 shows each model's language-family deviation (percentage points) relative to its own EN-safe mean.
• Ranking compares three schemes, all conditional on the model's own English safety: Scheme 1 is each model's own EN-safe set; Scheme 2 is the 39 prompts that are EN-safe for all ten models; Scheme 3b is the intersection of Claude-Sonnet-4's 95 prompts with each model's own set (Appendix H). A fourth construction that does not intersect with the model's own EN-safe set is excluded; the authors say it does not estimate the same quantity.

Experimental setup

• Evaluated models: Claude-Sonnet-4, Claude-Opus-4, Gemini-Pro-2.5, Gemini-Flash-2.5, GPT-5, Grok-4, Llama-Scout-17B, Llama-Maverick-17B, Mistral 2 Large in the main text (given as Mistral-Large in the tables), and Phi-4. The scoring model is GPT-4o.
• Data: 119 harmful prompts in 11 categories (Table 11); English plus 78 low-resource languages in 20 language families (Table 12). The full model has 9,401 rows, Grok-4 has 9,282 because Shan was not run; 93,891 pairs before filtering. 239,292 original responses; about 92,536 low-resource completions after filtering; 56,148 rows and 112 prompts of EN-safe language-level observations (Appendix F).
• Controls: No separate attack-method baseline. Fixed effects use Claude-Sonnet-4 as the reference. Scheme 3b for ranking also uses that model's EN-safe set as the baseline set.
• Metrics: Average unsafe percentage across 78 languages under EN-safe conditioning; proportion of prompts at ≥20% (≥16/78) and ≥50% (≥39/78); odds ratio of low-resource unsafety.
• Judging: Target-language responses are back-translated via Google Translate and labeled Safe or Unsafe by GPT-4o. The paper does not state a score threshold, nor report agreement with humans. Temperature is 0 where supported; the paper does not report the number of repeated samples.
• Spot check: 5 in-sample languages plus 1 out-of-sample language, 133 items each, from a separate generation (Section 3.4).

Main results

The average bypass rate range given in the abstract and Section 5.1 is 21.9% to 44.4%. The per-model numbers are taken from Rate 1 in Appendix H Table 6 and Scheme 1 in Table 5 (each model's own EN-safe set), not read off a figure. ORs are taken only from the values written in the Section 5.2 main text; Figure 4 has the remaining models, but the main text does not list numbers.

| Model | Scheme 1 bypass rate | EN-safe count | OR relative to Sonnet-4 | | Mistral-Large | 44.4% | 73 | 3.75 [3.11, 4.53] | | Phi-4 | 40.5% | 70 | 3.67 [3.04, 4.43] | | Llama-Scout-17B | 36.6% | 66 | not reported | | Claude-Sonnet-4 | 36.2% | 95 | 1 | | Claude-Opus-4 | 32.4% | 90 | not reported | | GPT-5 | 24.0% | 90 | 0.56 [0.47, 0.68] | | Gemini-Flash-2.5 | 21.9% | 60 | not reported |

Omitted from Table 6 Rate 1 / Table 5 Scheme 1: Llama-Maverick-17B is 27.4% (N=54), Grok-4 is 25.3% (N=53), Gemini-Pro-2.5 is 24.1% (N=70).

• The authors state that English safety often corresponds to a considerable proportion of unsafety in low-resource languages, and is not limited to a few languages. Figure 2 also plots the two breadth thresholds, but the main text does not list per-model values.
• The authors state that model differences remain large after fixed effects, and that the differences in Figure 2 cannot be explained only by the different EN-safe subsets of each model. None of the three OR intervals in the table contains 1.
• The authors also state that the order of this conditional rate cannot be directly treated as a publishable robustness ranking (Section 5.4).

Language-family structure

• On 56,148 rows, adding the model×language-family interaction gives ∆Deviance = 535.67 (63 df), dispersion-corrected p = 1.3 × 10−9. The authors state that the improvement is about 1.2% of the additive model deviance, small in magnitude but statistically significant; the language-family pattern is largely shared across models, with a small model-specific component (Section 5.3, Appendix F).
• The authors state that risk clusters in a few language families rather than isolated languages; under EN-safe conditioning the highest-risk language family is often Nilo-Saharan, and Niger-Congo is elevated for several models. Within the same model, language families differ by 9 to 34 percentage points (median 20); averaged over all models, the safest and highest-risk language families differ by about 16 percentage points. Figure 3 is a heatmap of deviations relative to each model's own mean, and the main text does not give per-cell numbers.
• The authors state that an attacker concentrating translation on the highest-risk language families would have a higher success rate than choosing languages at random; for mitigation, language-family-aware red-teaming and targeted fine-tuning can be prioritized.

Ranking under the three conditionalization schemes

• Scheme 1, Scheme 2 (39 common prompts) and Scheme 3b all condition on the model's own English safety. The authors state that they estimate the same attacker conditional quantity, only with different prompts entering the average. Table 7: Spearman ρ = 0.952 between Scheme 1 and 3b; 0.709 between Scheme 1 and 2; 0.636 between Scheme 2 and 3b. The main text writes the first pair as ρ = 0.95, and Scheme 2 with the others as ρ = 0.64–0.71.
• Nine models change order; only Mistral-Large has a rank span of 0. Claude-Sonnet-4 has a span of 5, entirely from Scheme 2, with rates of 36.2% under both Scheme 1 and 3b. Under Scheme 2, Mistral-Large is 19.5% and Gemini-Pro-2.5 is 5.8% (Table 6). The authors state that Scheme 2 is also the smallest set for each model, and its divergence is mixed with the smaller sample size and cannot be read alone as instability.
• The authors state that no scheme is suitable as a publicly released ranking: Scheme 1 has selection confounding; Scheme 3b is relative to one baseline model and is not neutral; Scheme 2 has a fixed prompt set but only 39 prompts. Therefore no leaderboard is released. The authors state that the models are not equally safe. Under the excluded fourth construction, only 53% to 83% of rows for non-baseline models are EN-safe for that model; Mistral-Large reaches 52.3%, and the authors state that part of the increase comes from the change in estimator.

Translation degradation and language-family interaction

• Four metrics fixed before viewing results flag 14.1% of EN-safe rows (main text); Table 8 gives 7,923 rows, 14.11%. Language-level degradation rate and bypass rate: main text Spearman ρ = 0.92 (p < 10−32, n = 78), appendix ρ = 0.921 (p = 6.23 × 10−33). Degradation rate ranges from 4.7% for Pashto to 34.0% for Baoulé. Degraded responses are judged unsafe 43.3% of the time, clean responses 29.8%. The authors state that the absolute per-language rates are contaminated to an unknown degree.
• After removing flagged rows (48,225 rows), the interaction remains: ∆Deviance = 440.87, 63 df, p = 3.3 × 10−6. Table 9 gives an attenuation of −17.70% (main text writes 17.7%). The binary control reduces 535.67 to 511.67 (Table 9 gives −4.48%, main text writes 4.5%); the continuous score gives −2.62% (main text 2.6%); entering the four metrics separately gives −6.40% (main text 6.4%). The authors state that the interaction remains significant under each control.
• The authors state that these analyses do not establish the language-family main effect, averaged over models, as independent of translation quality; what is established is the interaction, not the ranking. In Table 10, Shan has a bypass rate of 54.5% (n=662, degradation 32.2%), Dinka 50.0% (n=720); Pashto has a bypass rate of 24.0% and degradation of 4.7% (n=721). Grok-4 was not run on Shan.

Other ablations and analyses

• The fixed-effects sample is 681 pairs and 87 prompts; φ = 3.47, reproduced as 3.4742 in Appendix G. Prompts that are always 0 or always 78 are excluded.
• Appendix F language-family row counts: Other/Small 14,349, Indo-European 10,814, Austronesian 10,811, Niger-Congo 6,484, Turkic 4,323, Creole 3,604, Nilo-Saharan 2,882, Sino-Tibetan 2,881.
• In Table 11, the categories with the fewest items are Crimes Against Children and Sexual Violence & Illegal Sexual Acts, 1 each. The authors do not release by-category bypass rates.
• Appendix G: 23 of 24 checks reproduce exactly; the only difference is the model-level row counts. The main text writes 9,401 rows for the full model, 9,282 for Grok-4, and 93,891 before filtering. The appendix states that the original description conflates the two sources of 9,282, and retains that difference.
• The human spot check does not report the number of items per tier. The paper does not report which models do not support temperature 0, nor the number of repetitions.
• The examples in Appendix A are described by the authors as non-exhaustive and not statistically representative; the prompts and responses in it are not reproduced.

Limitations and future directions noted by the authors

• Translation-mediated evaluation: The pipeline uses one translation system to generate low-resource prompts and back-translates responses before English safety evaluation, which may introduce semantic drift. A spot check by professional translators supports prompt faithfulness, but does not cover the responses actually scored, nor provide an independent translation-quality measure for the full prompt set. The authors state that the measured rates should be interpreted as vulnerability under this translation-mediated threat model, not as a precise estimate of underlying multilingual safety behavior. A future direction is to validate the scored responses with bilingual human reviewers and an independent translation system (Section 6.2).
• Reviewer dependence: Unsafe labels are given consistently across all languages by a single LLM reviewer on translated outputs. The authors state that this favors scale and internal consistency, but may introduce systematic bias, including a possible dependence between the review policy and the behavior of the evaluated models (Section 6.2).
• Prompt and interaction scope: The harmful prompt set is curated and fixed in size, and the evaluation focuses on single turns. The authors state that real misuse can include iterative probing, multi-turn escalation and context accumulation, so the findings characterize a controlled, what the authors call conservative, translation bypass, not the full spectrum of multilingual adversarial interaction (Section 6.2).

Experimental coverage

• The evaluated models are Claude-Sonnet-4, Claude-Opus-4, Gemini-Pro-2.5, Gemini-Flash-2.5, GPT-5, Grok-4, Llama-Scout-17B, Llama-Maverick-17B, Mistral 2 Large (given as Mistral-Large in the tables), and Phi-4; the scoring model is GPT-4o (Section 3.3–3.4).
• 119 harmful prompts in 11 categories (Table 11); languages are English plus 78, in 20 language families, with resource tiers Joshi Level 0 and Level 1; Grok-4 was not run on Shan (Section 3.1–3.3, Table 12). Evaluation is single-turn prompt–response (Section 6.2).
• The main conditionalization uses each model's own EN-safe set; the paper also reports a 39-prompt ten-model common set, and the intersection with Claude-Sonnet-4 (Appendix H). The fixed-effects sample is 681 pairs and 87 prompts (Appendix E). The language-family test uses 56,148 rows (Appendix F).
• The translation spot check covers 5 in-sample languages plus 1 out-of-sample language, 133 items each, and comes from a separate generation (Section 3.4). The paper states that prompt-quality covariates were not recorded, and that there is no independent translation-quality measure for the full set (Section 5.5, 6.2).
• The main text gives the average bypass rate range, but does not list per-model values for the ≥20% and ≥50% thresholds in Figure 2. The OR numbers in the main text are only for Claude-Sonnet-4, Mistral-Large, Phi-4 and GPT-5 (Section 5.2). The by-category bypass ranking is not released (Section 1.4). The authors state that model-size effects are not separated (Section 3.3).

This is a measurement work: under the threat model of translation bypass, it separates English that is already unsafe from unsafe only on low-resource translations.

• Problem: Models are deployed globally, but safety evaluation and alignment are concentrated in English. The authors aim to measure whether harmful intents that are safely handled in English still receive unsafe completions after being switched to low-resource languages, whether risk is organized by language family, and whether a stable model ranking can be given.
• Approach: 119 English harmful prompts are translated via Google Translate into 78 low-resource languages, ten models respond in the target language, and the responses are back-translated and binary-classified by GPT-4o. The core quantity is the low-resource unsafe proportion conditional on that model's English safety, with two breadth thresholds of at least 16/78 and at least 39/78. Comparability uses prompt fixed effects, language family uses model×language-family interaction, and ranking uses three conditionalization schemes. The original prompts and the by-category ranking are not released.
• Bypass rate: On the own EN-safe set, the average bypass rate ranges from 21.9% for Gemini-Flash-2.5 to 44.4% for Mistral-Large (Table 6; the abstract gives the same range). Relative to Claude-Sonnet-4, the OR is 3.75 [3.11, 4.53] for Mistral-Large, 3.67 [3.04, 4.43] for Phi-4, and 0.56 [0.47, 0.68] for GPT-5 (Section 5.2, φ = 3.47).
• Language family and translation: Within the same model, language families differ by 9 to 34 percentage points (median 20). The interaction remains after removing degraded rows (Table 9 attenuation −17.70%). The authors state that the averaged language-family ranking was not established as independent of translation quality. Across 78 languages, the Spearman ρ between degradation rate and bypass rate is 0.92.
• Ranking: Nine models change order under the three schemes. The authors therefore do not release a leaderboard, while stating that the models are not equally safe.
• Authors' conclusion: If English alignment is treated as the safety boundary, translation is the simplest way to bypass it; English-only evaluation leaves a large searchable, unaudited multilingual surface. The authors argue for treating multilingual safety as safety infrastructure: measuring with conditional metrics aligned with attacker behavior, stress-testing by stable language groupings, and applying targeted mitigation where risk is concentrated.