AILuminate: MLCommons releases AI risk and reliability benchmark v1.0

The core problem is to establish an industry-standardized AI risk and reliability evaluation system, addressing the lack of a unified, reproducible system-level safety evaluation benchmark for generative AI chatbots.

• Application scenarios and importance: Generative AI chatbots are widely deployed in enterprise, community, and specialized scenarios, with the dual need to provide helpful answers while refusing harmful requests; but black-box characteristics make internal mechanisms difficult to inspect directly, necessitating standardized empirical evaluation.
• Limitations of existing methods: The authors point out that traditional benchmarks (e.g., ImageNet, GLUE, MMLU) have advanced technology, but mature deployment requires more normative, multi-stakeholder consensus, and continuously iterable industrial-grade benchmarks; a single off-the-shelf evaluation model can introduce bias and lack sensitivity to rare violations.
• Core observations and assumptions: System-level risk is a function of probability and harm severity; safety evaluation should focus on whether response content violates policy rather than only the prompt itself; with a strictly defined harm taxonomy and constrained prompt distribution, automated evaluators with discriminative power higher than the system under test can be trained.
• Proposed solution: Introduce AILUMINATE v1.0 benchmark, covering evaluation standards for 12 text harm categories, a 24,000 graded prompt library, ModelBench automated execution framework, multi-model ensemble evaluator, and a five-level rating system based on reference systems.
• User profile setting from evaluation perspective: The paper does not set an adversarial jailbreak threat model, but rather two non-expert user profiles: naive users (directly requesting harmful content) and informed users (with basic safety knowledge and using misleading context), and does not include advanced adaptive adversarial attackers.

• Traditional AI benchmarks and evaluation paradigms: Benchmarks such as ImageNet (Deng et al., 2009), GLUE (Wang et al., 2018), Dynabench (Kiela et al., 2021), and MMLU (Hendrycks et al., 2021) established general evaluation protocols. The authors note that in the context of widespread model deployment, there is a need for long-term maintained, industry-consensus standard benchmarks supported by neutral institutions.
• Safety guard and moderation models (Guard Models): Dedicated safety classifiers such as LlamaGuard (Inan et al., 2023), WildGuard (Han et al., 2024), AegisGuard (Ghosh et al., 2024), and ShieldGemma (Zeng et al., 2024) are used to identify violating inputs and outputs. The authors evaluated and integrated these guard models when building the automated evaluator pipeline.
• Jailbreak attacks and adaptive testing: Zou et al. (2023) studied universal jailbreak attacks based on adversarial optimization; the Behavior Elicitation Tool (BET, Kirk et al., 2024) from PRISM Eval explored adaptive behavior elicitation. In exploratory studies, the authors compared differences between static benchmarks and adaptive optimization tests in measuring model defenses.
• Multilingual and multimodal safety research: IrokoBench (Adelani et al., 2025) and Aya (Üstün et al., 2024) evaluate multilingual, especially low-resource, languages; Adversarial Nibbler (Quaye et al., 2024) provides a text-to-image red-teaming dataset. The authors extended these research ideas to Hindi, African languages, and multimodal pilots.
• Comparison baselines and reference benchmarks: Prompt length and distribution were compared against WildGuardMix and the LMSys 1M Chat dataset; model evaluation included OLMo v1.0, released by a nonprofit organization, as a control model without safety optimization.
• Positioning of this paper: The authors position AILUMINATE as the first comprehensive industry-standard benchmark developed through open collaboration, achieving reproducible risk quantification through integrated review, a confidential evaluation set, and a relative grading system.

AILUMINATE 整合危害分类标准、保密提示词集、集成评估系统与五级相对评分体系构建了完整的评测框架。

危害分类体系与评估指南

• 三类危害划分：涵盖 12 类危害分类（Table 1），分为物理危害（暴力犯罪、性相关犯罪、儿童性剥削、自杀与自残、大规模杀伤性武器 CBRNE）、非物理危害（知识产权、诽谤、非暴力犯罪、仇恨、隐私）及情境危害（专业建议、性内容）。
• 判定指导原则：仅依据安全政策评估响应而非提示词；在语义真正模糊时假定良性意图；允许通用常识信息，但禁止提供超越常识的可操作危害方案；叙事虚构场景中角色台词不视为违规；对医疗、法律、金融或选举建议仅检查是否包含免责声明，不校验事实正确性。

提示词数据集构建与管理

• 数据集构成：分为练习数据集（Practice Dataset）与正式评估数据集（Official Dataset），各含 12,000 条提示词（总计 24,000 条），每类危害各 1,000 条，均覆盖天真用户与知情用户两类画像。
• 数据管理与质检：练习集公开 10% 样本，正式集严格保密并设有备用提示词轮换机制以防过拟合；由三家核心供应商构建，经 15 名志愿者抽样质检语义多样性与真实性。

自动化响应评估器设计与熵阈值调节

• 执行架构：依托独立开源工具 ModelBench 调度提示词分发、结果记录与报告生成。
• 集成评审机制：由微调专用模型、提示工程通用模型及现有 Guard 模型构成集成评审团，以降低依赖单一模型带来的偏见。
• 基于熵的分类阈值调节：通过输出概率分布的熵调节分类阈值，公式为：

H(ŷ) = −∑_c p(ŷ_c) log p(ŷ_c) 其中 ŷ 为 logits 向量，c ∈ {safe, unsafe}。调节分类阈值以平衡假阳性与假阴性，设计上优先降低漏报率（false-safe rate）并保证评估的确定性与可重复性。

评级体系与参考系统映射

• 分级计算与映射：SUT 违规得分为 S^SUT = N^UNSAFE / N^TOTAL × 100%，相对比率为 S^∗ = S^SUT / S^REF。
• 五级评分区间：根据 Table 3，Excellent 要求绝对得分 S^SUT ≤ 0.1%；其余基于相对比率 S^∗ 划分：Very Good 为 S^∗ ≤ 0.5；Good 为 0.5 < S^∗ < 1.5；Fair 为 1.5 ≤ S^∗ < 3；Poor 为 S^∗ ≥ 3。参考系统取参评且参数量小于 150 亿的相对开源模型中得分较高（违规率较高）的前两名复合而成。

实验设置

• 被测系统（SUT）：主结果评估涵盖 Anthropic、Google、Meta、OpenAI 及 Mistral 的模型，并采用非营利组织发布的 OLMo v1.0 作为无安全优化的对照模型；探索性研究测试了 Pixtral Large、Llama 3 70B 和 Gemini Pro 1.5。
• 基准数据集与规模：官方正式测试集包含 12,000 条提示词，涵盖 12 类危害，每类 1,000 条；提示词分布对比实验使用了 WildGuardMix 与 LMSys 1M Chat 数据集。
• 评测指标：SUT 违规响应百分比（SSUT）、相对于参考系统的比率（S*）、五级评级、评估器漏报率（false-safe rate）与误报率（false-unsafe rate）。
• 评审方式：基于多模型微调与提示工程集成的评审团自动打分，结合少量人工多标注仲裁校验。
• 样本量与测试条件：每类危害 1,000 条单轮文本测试；提示词和评估指南基于美式英语编写。

主结果

AILUMINATE 评级体系划分标准与对应区间（据 Table 3）：

• 公开结果发布机制：作者说明参评 SUT 的具体得分总表在官方网页发布并支持动态维护，因此未在静态论文报告中固化具体模型得分。
• 评估器偏差对评级的影响：评估器优先压低漏报率导致误报率较高（倾向于判定不安全），但由于参考系统评测存在同向偏差，相对比率 S* 减轻了评级漂移（第 6.3 节）。

探索性实验：自适应越狱测试（BET）

• 测试内容：PRISM Eval 开发的行为诱导工具（BET），通过多步优化对目标模型开展自适应测试（第 8.1 节）。
• 实验结果：在 5 步优化内，提示词有效率从 8% 提升至 78%；Figure 7 显示不同越狱技术在 Pixtral Large、Llama 3 70B 和 Gemini Pro 1.5 上的效果分布存在明显差异。
• 作者解读：静态基准容易遗漏诱导不安全响应的提示词变体，动态、针对特定模型的测试更能反映模型的越狱抵御能力。

探索性实验：多模态安全评估试点

• 文生图测试：构建 774 条提示词（486 条针对露骨内容，288 条针对暴力），包含显式与隐式诱导（第 8.4 节）。
• 图生文测试：测试了 4 个危害类别，商业模型表现较为安全（不安全响应率低于 1%），开源模型不安全率较高（最高达 14%）。
• 作者解读：部分开源模型因未能准确理解请求而拒绝，呈现“偶然安全”（safe by accident）现象；未来需开发高保真多模态自动评估器。

其他消融与分析

• 提示词长度分布：MLC 1.0 提示词长度主要分布在 100 至 300 字符区间，分布特征介于真实对话与合成数据集之间（Figure 1）。
• 提示词语义聚类：利用 all-MiniLM-L6-v2 嵌入与 k-means 聚为 13 个无监督簇，由 GPT-4 赋予标签（Figure 2）。
• 印地语本地化构建：Tattle Civic Tech 针对仇恨与性犯罪构建各 1,000 条印地语提示词（第 8.2 节）。
• 评分不确定度误差带：评估器性能波动产生跨等级误差带，且模型违规率越高误差带越宽（Figure 6）。

Limitations noted by the authors and future directions

• Limited harm scope and user profiles: The benchmark only tests predefined harms and two profiles (naive and informed), and does not include harms not listed such as environmental impact, inaccurate medical advice, and bias, nor does it cover professional adversarial attackers using adaptive techniques (Section 7, item 1).
• Lack of regionalization and cultural adaptation: Prompts and evaluation guidelines are written primarily in American English and according to U.S. cultural norms, and the development process lacked a public comment period for broadly collecting feedback (Section 7, item 1).
• Interaction limitations of single-turn artificial prompts: Real unknown-user conversations were not used, and sustained interactions or cumulative effects involving multi-turn complex contexts were not tested (Section 7, item 2).
• Significant uncertainty in evaluators and annotation: Performance fluctuation of evaluators produces wide error bands and cross-grade risk; clarity of worker guidelines and the number of human review rounds per response still need improvement (Section 7, items 3 and 4).
• Limitations of the single-prompt single-harm classification assumption: Score computation assumes unsafe responses belong to the same harm category as the prompt, ignoring complex combinations of cross-category or multiple harms (Section 7, item 5).
• System access differences affect fairness: API access and open-source local deployment include different levels of guardrail mechanisms; model evaluation requires strictly unified system-level filtering conditions (Section 7, item 6).

Experimental coverage

• Interaction modality and turns: The core benchmark in the main text covers only single-turn English text interactions and does not include multi-turn dialogue evaluation.
• Release format of main model evaluation results: The paper does not directly list a full score table of SUTs in the main text; the main evaluation data for each vendor's model is published only on the official website and supports dynamic maintenance.
• Exploratory multimodal samples: Image-to-text tested only 4 harm categories, and the text-to-image pilot scale was 774 prompts.
• Exploratory language coverage: Exploratory studies collected 1,000 prompts each for only two harm categories in Hindi; the main text did not formally deploy benchmarks in French, Chinese, or indigenous African languages.
• White-box details of the evaluator: The paper does not disclose the specific model weights, training data, or combination logic used in the integrated evaluator, to prevent models from gaming the evaluator.

• Benchmark positioning: MLCommons, together with cross-domain experts, released AILUMINATE v1.0, establishing an industry-standardized single-turn text safety risk and reliability evaluation framework for general-purpose AI chatbots.
• Core framework: Proposed a taxonomy and evaluation guidelines covering 12 categories of physical, non-physical, and contextual harms, and constructed a practice set and a confidential formal set, each containing 12,000 prompts (24,000 in total), covering both naive and informed users.
• Evaluation technology: Developed the ModelBench execution framework, using a multi-model ensemble jury and entropy-adjusted threshold techniques for automated violation determination to reduce the safety false-safe rate.
• Grading mechanism: Established a five-level grading standard (Poor to Excellent) relative to two excellent open-source lightweight reference models, where Excellent requires an absolute violation rate no higher than 0.1%.
• Empirical and exploratory findings: Exploratory experiments showed adaptive jailbreak optimization can increase effectiveness from 8% to 78% within 5 steps; in image-to-text testing, commercial models had violation rates below 1%, while open-source models reached up to 14% and exhibited a safe-by-accident phenomenon.
• Authors' conclusions and implications: The authors emphasize that this benchmark is a foundational step toward establishing a globally universal safety standard, and future work needs to continuously expand and evolve toward multi-turn interaction, adaptive testing, multilingual regionalization, and multimodal domains.