ScopeBench: Evaluating LLMs' awareness of cultural norm scope
ScopeBench: Evaluating Cultural Norm Scope Awareness in LLMs
Translation in progress — untranslated text is omitted.
- Existing safety evaluations treat harm as culturally invariant, while cultural evaluations treat the scope of applicability of norms as given; neither can measure whether a model knows whom a norm constrains. The authors argue that misjudgments in both directions (imposing specific norms, relativizing universal protections) are safety failures.
- Construct ScopeBench: 3,323 norms (975 UNIVERSAL, 2,348 SPECIFIC), confirmed by majority annotation of three annotators; define three types of scope errors, SI/SD/SP, and evaluate 21 LLMs on four tasks—classification, joint attribution, item-by-item judgment, and behavioral interaction—using the same operational definition.
- The authors report: the dominant error type reverses with prompt format (Task 1 is dominated by Deflation, Task 2b by Inflation); six models with Task 1 accuracy ≥97% still have 40.3%–42.4% harmful responses in Task 3; HRtotal is 29.0%–51.2% (excluding one systematically refusing model).
- Limitations noted by the authors: English-only evaluation; Task 3 relies on GPT-5.5 automated judging, and all violation statements are produced by the same generator. Experimental coverage: 21 models, zero-shot setting; human validation is on a stratified subsample (200 pairs for Task 3), and cross-judge review is a sample of 500 per model.