Skip to content

ScopeBench: Evaluating LLMs' awareness of cultural norm scope

ScopeBench: Evaluating Cultural Norm Scope Awareness in LLMs

arXiv: 其他 AI 安全论文· arXiv:2611.18600· Haojun Liang, Ziwei Zhao, Chenyu Li, Qiyue Zhao, Rongduomijia Mu, Yulan Shen, Yu Gu, Chen Zhang, Kairui Zhang, Yanbiao Ma, Wenxuan WangTechnical University of Munich; University of Texas at Austin; Chongqing University; University of Illinois at Urbana-Champaign; Renmin University of China; renlab.aiIndexed by this site · 2 days ago· Originally publishedEvaluation· Its place in the scan record →

Translation in progress — untranslated text is omitted.

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

Questions
Existing safety evaluations treat harm as culturally invariant, while cultural evaluations treat the scope of applicability of norms as given; neither can measure whether a model knows whom a norm constrains. The authors argue that misjudgments in both directions (imposing specific norms, relativizing universal protections) are safety failures.
Methods
Construct ScopeBench: 3,323 norms (975 UNIVERSAL, 2,348 SPECIFIC), confirmed by majority annotation of three annotators; define three types of scope errors, SI/SD/SP, and evaluate 21 LLMs on four tasks—classification, joint attribution, item-by-item judgment, and behavioral interaction—using the same operational definition.
Experiments and results
The authors report: the dominant error type reverses with prompt format (Task 1 is dominated by Deflation, Task 2b by Inflation); six models with Task 1 accuracy ≥97% still have 40.3%–42.4% harmful responses in Task 3; HRtotal is 29.0%–51.2% (excluding one systematically refusing model).
Limitations and future work
Limitations noted by the authors: English-only evaluation; Task 3 relies on GPT-5.5 automated judging, and all violation statements are produced by the same generator. Experimental coverage: 21 models, zero-shot setting; human validation is on a stratified subsample (200 pairs for Task 3), and cross-judge review is a sample of 500 per model.
Read originalarxiv.org