IndicSafe: A Multilingual LLM Safety Benchmark for 12 Indic Languages in South Asia

The same English prompt often receives inconsistent safety labels when translated into different Indic languages.

• Scenario: The authors state LLMs are already used in safety-critical multilingual applications such as content moderation and civic platforms, but most safety evaluations focus on English or high-resource Western languages, lacking cultural specificity.
• Gap: Speakers of 12 Indic languages exceed 1.2 billion, and the authors state they are underrepresented in pretraining corpora and safety data; caste, religion, gender, and politics are often absent from English safety benchmarks. The authors state existing non-English evaluations often lack regional settings or rely on machine translation.
• Observation: The authors use a caste-supremacy question as an example (original sentence not reproduced): English and Hindi are labeled UNSAFE, while Odia, Telugu, and Punjabi are labeled SAFE or AMBIGUOUS. The authors state that in deployment, users may bypass safety controls simply by switching languages.
• Assumption: The authors state existing alignment assumes cross-language consistency by default, but this consistency does not hold in practice; safety alignment is not language-invariant. The introduction states that over 45% of harmful prompts receive inconsistent cross-language safety labels; the results section does not provide a corresponding calculation table.
• Proposal: The authors build 6,000 culturally grounded prompts, translated by native speakers into 12 languages, evaluate 10 LLMs, and quantify drift and refusal bias using cross-language agreement rates, category bias, and prompt-level entropy. The authors state IndicSafe is the first safety benchmark with cultural information for Indic deployment, and advocate for language-aware alignment targeting regional harms.

English Safety and Toxicity Evaluation

• Toxicity and refusal: RealToxicityPrompts (Gehman et al., 2020), IndicJR (Pattnayak and Chowdhuri, 2026), and DetoxLLM (Khondaker et al., 2024) are classified by the authors under toxicity evaluation; Ouyang et al. (2022) and Bai et al. (2022) are classified under instruction following and refusal.
• Red teaming: RED Teaming (Ganguli et al., 2022) extends the above work with adversarial prompts. The authors state that it remains predominantly monolingual and Western-centric.
• Fairness evaluation: HolisticEval (Bommasani et al., 2023), SweEval (Patel et al., 2025), and FairEval (Sah et al., 2025) are said by the authors to involve fairness, but still lack cross-lingual and culturally adaptive evaluation, thus limiting global applicability.

Multilingual and Low-Resource Safety

• XSafety (Wang et al., 2024): Evaluates instruction-style safety prompts across 10 global languages. The authors state that performance drops outside English; prompts are not region-specific, and they are machine-translated without analyzing safety drift across translations.
• IndoSafety (Azmi et al., 2025): Uses culture-related prompts to examine harms in Indonesian. The authors state it is language-specific, not generalizable to Indic, and does not focus on drift across translations. Table 1 marks its human translations as Partial.
• Afrocentric NLP (Adebara and Abdul-Mageed, 2022): The authors list it as a similar concern without elaborating on methodological comparisons.

Indic Resources and Benchmark Comparisons

• PARIKSHA (Watts et al., 2024): Table 1 records 10 Indic languages, human-translated; cultural context, safety orientation, and cross-lang drift are all marked as no.
• Samanantar (Ramesh et al., 2023): Table 1 records 11 Indic languages as human-translated resources; similarly marked as not safety-oriented and with no drift analysis. The main text does not elaborate on these two works' experimental protocols.
• Baselines and data: The paper does not use existing detectors or the above benchmarks as scoring baselines. The experimental comparison is among 10 LLMs on the authors' own IndicSafe; Table 1 only compares benchmark properties.

The authors state that this paper addresses Indic safety issues rooted in local cultural, social, and political harms, using 6,000 prompts translated into 12 languages by native speakers, along with analyses of cross-language consistency, refusal bias, category bias, and prompt-level entropy, to point out weaknesses in safety generalization. In the contribution list and Table 1 caption, the authors state that only this work simultaneously has cultural grounding, native-speaker translation, safety orientation, and drift analysis.

The authors built IndicSafe as a diagnostic benchmark: English prompts serve as the reference, and the question is whether the 12 translations of the same prompt receive different safety labels. Models are treated as black-box via API, with no internal visibility.

Prompt Design and Categories

• Researchers with regional, sociolinguistic, and political backgrounds handwrote prompts in English, drawing material from real-world discourse, news, and harmful online content. There are 500 items per language, 6,000 across 12 languages.
• The nine categories in Table 2 and item counts per language: Gender & Caste Discrimination 45; Harmful Instructions 51; Harmless Control Prompts 125; Health Misinformation 45; Misinformation or Conspiracy 50; Offensive / Hate Speech 49; Political Manipulation 45; Religious Hate or Provocation 45; Tricky / Ambiguous Prompts 45.
• Forms include statements, questions, and commands. Categories derive from qualitative analysis of Indian-language online discourse, regional media controversies, legal speech boundaries, and South Asian sociolinguistic harm research. Specific harmful original sentences are not reproduced.
• The authors state the goal is to simulate culturally grounded adversarial inputs that may appear in Indic deployment, rather than only English prompts or direct translation.

Translation and Language Coverage

• Languages are Hindi, Bengali, Odia, Tamil, Telugu, Kannada, Malayalam, Marathi, Gujarati, Punjabi, Nepali, and Urdu, spanning Indo-Aryan and Dravidian. The authors state speakers exceed 1.2 billion, but these remain low-resource relative to English: speaker population does not equal high-quality digital text or safety benchmarks.
• Native speakers translated per Appendix A.1, with requirements for cultural and contextual equivalence, preserving intent, tone, and harm framing, without literal translation or arbitrary removal of harmful intent. 50–100 items per language underwent bilingual review.
• Table 7: 720 items reviewed in total, 28 replaced. A total of 18 native speakers and reviewers were involved on the translation side. Untranslatable items were replaced or removed from that language set. Figure 10 covers corresponding languages in India and Pakistan, Bangladesh, Nepal, and Sri Lanka, excluding Dhivehi and Dzongkha.

Generation Protocol

• Each of the 10 models ran 6,000 items, totaling 60,000 generations. Unified settings: max 200 tokens, temperature 0.3, top_p 0.75, no penalty.
• The main text states Claude, GPT-4o, and Grok-3 use role-based formats, while LLaMA, Mistral, and Cohere use open instructions. The corresponding name in the model list is GPT-4o Mini, not the GPT-4o used for review.
• Prompts are sent in native script, without translation prompts, labels, or examples. Appendix A.3 / Table 8 give model specifications and formats.

Safety Judgment

• The review model is GPT-4o, temperature 0.0, max 150 tokens. Each response is labeled SAFE, UNSAFE, REFUSAL, or AMBIGUOUS, with an explanation. The authors state the judgment considers the response's content, tone, and intent, independent of input language.
• Appendix A.5 human definitions: SAFE is factually correct, harmless, or a clear and appropriate refusal; UNSAFE is harmful, offensive, biased, manipulative, misleading, or toxic; REFUSAL is an explicit refusal without additional unsafe or misleading content; AMBIGUOUS is vague, self-contradictory, incomplete, or unclear in intent. Example original sentences are not reproduced. The review prompt is given as a figure (Figure 4; also called Figure 5 in Appendix A.4); the text conversion does not include the prompt text.
• The input scope for human review and automated review differs: automated judgment, per the authors, is independent of input language and focuses on the response; the human guidelines require looking at both prompt and response.
• The human sample is a 5% stratified sample of 3,000 generations, covering categories, languages, and models. 18 bilingual annotators used guidelines aligned with the review prompt. A subset was cross-annotated, with a senior reviewer adjudicating.

Metrics

• Cross-Language Agreement Rate: Whether all translations of the same prompt receive the same label for the same model.
• Cross-Model Agreement Rate: Whether all models give the same label for the same language.
• Majority Agreement on Harmful Prompts: Whether prompts harmful in English are labeled harmful by the majority of models or languages.
• The rest are Judgment Entropy, Ambiguity Rate, Language Consistency Index (average entropy across languages; lower means more consistent), standard deviation of Intra-Model SAFE%, Category Bias Score, and Cohen's κ between GPT-4o and humans. Definitions are in Table 3. §6.2 states LCI of 0 means labels are identical across languages, and 2 means the four label categories are evenly split. The paper does not provide formulas for entropy or the bias score.

Experimental Setup

• Generation models: GPT-4o Mini, Claude Sonnet v4, Grok-3 (sizes all listed as Unknown); Meta LLaMA 4 (17B), LLaMA 3.3 (70B), LLaMA 3.1 (405B); Cohere Command R (paper says released 2024-08), Cohere Command A (released 2025-03), sizes not given for either; Qwen1.5-7B-Chat (7B); Mistral-7B-Instruct-v0.2 (7B). All via API, black-box.
• Data: IndicSafe, 500 items in each of 12 languages, 6,000 in total; distribution across nine categories in Table 2. No independent baseline detectors; the 10 models serve as comparisons against each other.
• Generation: 6,000 items per model, 60,000 generations in total. Max 200 tokens, temperature 0.3, top_p 0.75, no penalty. The paper does not report repeated sampling.
• Review: GPT-4o performs four-way classification, temperature 0.0, max 150 tokens; the paper gives no numeric thresholds. Human review on 3,000 items (5%), 18 annotators.
• Additional checks: Error analysis on 400 items; 5,000 items tokenized with IndicBERT and XLM-R. Tokenization models do not enter the safety results tables.

Main Results

Table 4: Across all 60,000 generations, SAFE 50.5%, UNSAFE 18.7%, REFUSAL 18.4%, AMBIGUOUS 12.4%. The table below is the model breakdown from Table 5 (in %). Only 8 rows can be listed for the 10 models; LLaMA 3.1 and LLaMA 3.3 are omitted.

Swipe horizontally to read the table

Model	SAFE	UNSAFE	REFUSAL	AMBIG.
Grok-3	83.67	0.98	12.40	3
LLaMA 4	78.72	5.28	12.18	3.8
Claude Sonnet4	60.38	2.63	27.08	10
GPT-4o Mini	58.53	20.70	16.05	4.7
Cohere A	59.20	12.93	8.77	19.1
Cohere R	33.82	33.53	11.18	21.5
Mistral-7B v0.2	12.85	45.50	35.30	6.4
Qwen 1.5	4.55	49.52	15.98	30

• The two rows from Table 5 not listed above: LLaMA 3.1 is 56.90 / 7.60 / 22.10 / 13.4; LLaMA 3.3 is 56.42 / 8.72 / 22.70 / 12.2 (same order: SAFE, UNSAFE, REFUSAL, AMBIG.).
• The authors state that Grok-3 and LLaMA 4 give SAFE more often, with lower ambiguity and refusal; Qwen and Mistral give UNSAFE more often. The main text says their SAFE rates are 84% and 79%, while Table 5 gives 83.67 and 78.72.
• The authors state that Claude adopts a high-refusal strategy (main text says 27%, Table 5 gives 27.08), and GPT-4o Mini is more balanced between safety and expressiveness. The authors interpret these differences as reflecting different providers' safety-tuning paradigms; the paper provides no training or alignment details to verify this interpretation.

Cross-Language and Cross-Model Drift

• §6.2: Cross-Language Exact Agreement Rate for the same prompt and same model is 12.8%; cross-language majority agreement for harmful prompts is 63.3%. The authors say that under exact matching nearly 90% diverge, and write that identical prompts diverge ~90% of the time. For the same prompt and same language, cross-model exact agreement is 0.35%, and cross-model majority agreement for harmful prompts is 54.4%.
• The Cross Language Consistency in Table 6 is not the same column as the previous numbers: Grok-3 is 84.8 (SAFE variance 7), Mistral 7bv0.2 is 58.0 (variance 11), and Cohere Command A's SAFE variance is 14, the highest in the table. The main text says Grok-3 has 85% consistency. The abstract says SAFE rate variance exceeds 17%, and the Discussion says reaching 17%; Table 6 has no cell with 17, and the paper does not explain the relationship between the two, nor how Table 6's Consistency corresponds to 12.8%.
• LCI: Grok-3 is 0.51, Mistral-7Bv02 is 1.35 (Figure 1; the main text gives only these two numbers). The authors state that safety judgments are not language-invariant. The authors state that Claude and Grok-3 lean more toward refusal, while Qwen and Mistral are more likely to produce unsafe or ambiguous output. The appendix says that in Figure 9, Claude and GPT-4o have the highest agreement rate, and Qwen and Mistral diverge most from other models; cell values are not given in the main text, and the evaluation list reads GPT-4o Mini.

Language, Refusal, and Ambiguity

• The authors state that in Figure 2, Odia, Kannada, and Punjabi have higher AMBIGUOUS and REFUSAL, while Hindi and Marathi have higher SAFE and lower ambiguity; per-language percentages are not given in the main text. The Discussion says that Odia and Punjabi have AMBIGUOUS up to 28%, and GPT-4o-mini's SAFE > 60% on Hindi and Marathi. The main text for Figure 7 says Cohere Command R, Command A, Claude Sonnet 4, and GPT-4o-mini have lower SAFE on Odia, Gujarati, and Kannada; cell values are not given. The caption for Figure 8 says Odia, Punjabi, Gujurati, and Telugu have high REFUSAL.
• The highest refusal rate for harmless prompts is 61.6% (Mistral-7Bv02, Punjabi, Table 11). The authors say Mixtral and Qwen refuse harmless prompts more often in Tamil, Odia, and Bengali; the evaluation list reads Mistral-7B-Instruct-v0.2, with no explanation for Mixtral. The authors say that for Political and Religious, over 60% of responses are labeled unsafe or refusal; Health and Caste are relatively more balanced. The discussion of Figure 6 says Political Manipulation, Harmful Instructions, and Religious Provocation have the highest REFUSAL, Tricky/Ambiguous is mainly AMBIGUOUS, and Harmless Control is mostly SAFE, with 4–9% still refused.
• §6.4 says the highest ambiguity rate is Qwen (Odia) at 53.8%. However, the title of Table 13 says it is the proportion of AMBIGUOUS among human annotations, and the highest row is also qwen1.5-7B / Odia at 53.80. 11.2% of prompts have entropy > 1.8, with the top 5 at 2.0; the Discussion writes over 11%. The authors say that 80% of ambiguous cases in annotation notes contain hesitant or uncertain expressions, without giving the denominator.

Human Agreement and Errors

• §4.3: Average Cohen's κ between GPT-4o and humans is 0.64 for categories and 0.63 for languages. Table 9 extremes: categories range from 0.56 for Tricky / Ambiguous Prompts to 0.71 for Harmful Instructions; languages range from 0.57 for Nepali to 0.69 for Hindi. §4.2 writes κ≈0.64 and weighted κ≈0.67.
• After weighting, the overall value goes from 0.64 to 0.67. The authors say Misinformation goes from 0.59 to 0.71, and Ambiguous from 0.56 to 0.61; they give CIs: Misinformation +0.122 (+0.05–0.21), Ambiguous +0.05 (+0.01–0.17), calling these significant gains. §5.6 also writes a high 0.67 inter-annotator agreement, without giving a calculation table; it should not be equated directly with the weighted κ.
• Appendix A.14's 400 items: 33 false negatives (8.25%), 15 false positives (3.8%), and 11 over-refusals (the paper writes 2.1%). The authors say harmless prompts are sometimes refusedibles, and harmful outputs in low-resource languages are missed; these 400 items are not broken down by language for FN counts. The authors believe tokenization UNK/OOV is unlikely to explain the drift.

Other Ablations and Analyses

• Table 11: The top two harmless refusal rates are Mistral-7Bv02–Punjabi at 61.6% and qwen1.5-7B–Hindi at 33.6%; the lowest row listed in the table is qwen1.5-7B–Bengali at 21.6%.
• Table 13: 53.80 (Odia) and 40.40 (Malayalam) are at the top; another row reads qwen1.5-7Ben–Tamil at 34.80.
• Entropy table: The top 5 in the appendix are 2.00, 2.00, 1.96, 1.96, 1.96, with model names qwen_outputs, llama33, CohereR, mistral7bv02; the original prompts are not reproduced.
• Table 10: Misinformation's quadratic weighted κ is 0.712, linear weighted is 0.672, and unweighted is 0.59.
• Table 14: On 5,000 items, both tokenizers have UNK/OOV at 0.00%; average tokens per word are 1.325 vs 1.469. The authors write that token counts are about 10–11% higher, with about +0.14–0.15 per word.
• Table 7: 720 items reviewed and 28 replaced; Tamil has 5 replacements out of 60 items, the highest replacement count in the table.

Limitations and Future Directions Noted by the Authors

• Translation: Translated prompts may still have subtle meaning shifts (Limitations).
• Automated Review: GPT-4o-based automated evaluation may miss subtle or culturally specific harms (Limitations).
• Analysis Target: Only model outputs are analyzed, not training data or internal mechanisms (Limitations).
• Interaction Length: Only single-turn prompts are used; multi-turn and long-text safety are left for future work (Limitations).
• Future Claims: The Conclusion proposes incorporating culturally diverse prompts, multilingual alignment objectives, and prompt-level fluctuation metrics into future safety tuning. This is the authors' proposal, not a completed release or integration.

Experimental Coverage

• The generation models tested are the 10 in §4.1 and Table 8, all accessed via API and treated as black-box; safety labels are provided by GPT-4o.
• The evaluation data is the self-built IndicSafe: 500 items per language across 12 languages, 6,000 in total, with nine categories in Table 2; 60,000 generations in total. XSafety, IndoSafety, PARIKSHA, and Samanantar discussed in Table 1 do not appear in the result score tables.
• Human review covered a stratified sample of 3,000 items with 18 bilingual annotators; error analysis covered 400 items; tokenization checks covered 5,000 items with UNK/OOV at 0.00%.
• Generation settings: max 200 tokens, temperature 0.3, top_p 0.75; review temperature 0.0, max 150 tokens. The number of generations matches 10×6,000. The paper does not report repeated sampling.
• Experiments used single-turn prompts. The Category Bias Score is summarized in §6.3 as over 60% of Political and Religious items labeled unsafe or refusal; the paper does not report a per-model table for this score.

IndicSafe uses culturally grounded translations to check whether safety labels of 10 LLMs change across 12 Indic languages.

• Problem: The authors state that safety evaluation mostly focuses on English or high-resource Western languages, while harms related to caste, religion, gender, health, politics, etc., lack corresponding evaluation in South Asian contexts. These languages, spoken by over 1.2 billion people, are underrepresented in training and safety data.
• Approach: Handwritten English prompts translated by native speakers into 12 languages, 500 per language, 6,000 total, including harmless controls. Each model generated once via black-box API, 60,000 total; GPT-4o labeled as SAFE, UNSAFE, REFUSAL, or AMBIGUOUS. The mean Cohen's κ for 3,000 human-annotated categories is 0.64.
• Overall labels: In Table 4: SAFE 50.5%, UNSAFE 18.7%, REFUSAL 18.4%, AMBIGUOUS 12.4%. In Table 5: Grok-3 SAFE 83.67, Qwen 1.5 UNSAFE 49.52, Mistral-7B v0.2 UNSAFE 45.50 and REFUSAL 35.30.
• Drift: Exact cross-language agreement for the same prompt and model is 12.8%; cross-language majority agreement for harmful prompts is 63.3%; exact cross-model agreement for the same prompt and language is 0.35% (§6.2). The authors state that under exact matching, nearly 90% disagree. Table 6 lists a separate Consistency, with Grok-3 at 84.8; the paper does not explain its relation to 12.8%. The abstract states SAFE variance exceeds 17%, while the highest value listed in Table 6 is 14.
• Refusal and misses: Highest refusal rate for harmless prompts is 61.6% (Mistral-7Bv02, Punjabi). In the 400-item error analysis, false negatives are 8.25% and false positives 3.8%. The authors consider tokenization missing characters unlikely to explain the drift.
• Author conclusions: The authors state that English safety alignment cannot be assumed to transfer to these multilingual, culturally grounded scenarios; the gap involves both language and socioculture. They advocate for language-aware alignment targeting regional harms and for incorporating prompt-level variability into future safety tuning.