MLCommons releases AI Safety Benchmark v0.5

The MLCommons AI Safety Working Group releases AI Safety Benchmark v0.5 for evaluating the safety risks of AI systems using dialogue-fine-tuned language models. This version covers only a single use case (an adult chatting with a general assistant in English) and a limited set of personas (ordinary user, malicious user, vulnerable user), and proposes a taxonomy containing 13 hazard categories, of which 7 have tests in v0.5, with a total of 43,090 test items generated from templates. The benchmark also includes a scoring system, the open ModelBench evaluation platform and downloadable tools, and an example report evaluating more than ten publicly available dialogue-fine-tuned language models. The authors plan to release v1.0 by the end of 2024, and explicitly state that v0.5 should not be used to evaluate the safety of AI systems, while fully documenting the limitations, flaws and challenges of this version.