Holistic safety and responsibility evaluations of advanced AI models

Google DeepMind publishes a report summarizing a set of safety and responsibility evaluation methods it applies in the development of advanced AI models, and shares lessons with a broader audience. The report presents three main lessons: a theoretical framework helps organize the breadth of risk domains, modalities, forms, metrics, and objectives; both the theory and practice of safety evaluation benefit from collaboration to clarify goals, methods, and challenges and to convey insights across different stakeholders and disciplines; similar key methods, lessons, and institutions apply to various issues of responsibility and safety, including established and emerging harms, so the safety evaluation and safety research communities should collaboratively develop, refine, and implement new evaluation methods and best practices rather than operating in isolation. The report concludes by noting the need to rapidly advance the science of evaluation, integrate new evaluations into AI development and governance, establish scientifically grounded norms and standards, and foster a robust evaluation ecosystem.