Improving evaluation methods for large language models across global languages

Led by Singapore AISI, participants from Singapore, Japan, Australia, Canada, the EU, France, Kenya, South Korea, and the UK in the International Network for Advanced AI Measurement, Evaluation and Science conducted a joint multilingual safety evaluation of two open-weight models, covering ten languages: Cantonese, English, Persian, French, Japanese, Korean, Swahili, Malay, Mandarin, and Telugu. The evaluation used more than 6,000 newly translated prompts, covering five harm categories: privacy, non-violent crime, violent crime, intellectual property, and jailbreak robustness, and employed both LLM-as-a-judge and human annotation as review methods. The results show that model safety behavior differs across languages, guardrail robustness varies by language and harm type, and the reliability of LLM review and human review is inconsistent. This work also summarizes methodological points for improving multilingual safety evaluation, such as the need for translation that incorporates cultural context, stress-tested review prompts, and clearer human annotation guidelines, and describes this as an initial step toward a shared framework for multilingual safety testing of advanced AI systems.